337 lines
8.2 KiB
Markdown
337 lines
8.2 KiB
Markdown
# Milestone 19: Security Testing Suite
|
|
|
|
**Status**: Complete
|
|
**Goal**: Comprehensive security test coverage with fuzzing.
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
This milestone formalizes the security testing infrastructure:
|
|
- Unit tests for all sandbox components (already implemented in Milestones 1-18)
|
|
- Integration tests for full app lifecycle
|
|
- Fuzzer for random Lua code testing
|
|
- Security audit checklist verification
|
|
|
|
### Key Deliverables
|
|
|
|
1. **LuaFuzzer class** - Generates random Lua code and verifies sandbox integrity
|
|
2. **Integration tests** - Full lifecycle tests
|
|
3. **Security audit tests** - Verify all SANDBOX.md security requirements
|
|
|
|
---
|
|
|
|
## File Structure
|
|
|
|
```
|
|
sandbox-test/
|
|
├── src/
|
|
│ ├── main.cpp # Existing - all unit tests (135+)
|
|
│ ├── lua_fuzzer.h # NEW - Fuzzer header
|
|
│ └── lua_fuzzer.cpp # NEW - Fuzzer implementation
|
|
```
|
|
|
|
---
|
|
|
|
## Implementation Details
|
|
|
|
### 1. LuaFuzzer Class
|
|
|
|
```cpp
|
|
// lua_fuzzer.h
|
|
#pragma once
|
|
|
|
#include <string>
|
|
#include <vector>
|
|
#include <random>
|
|
#include <functional>
|
|
|
|
namespace mosis {
|
|
|
|
struct FuzzResult {
|
|
bool crashed = false;
|
|
bool sandbox_intact = true;
|
|
std::string error;
|
|
size_t iterations = 0;
|
|
size_t errors_caught = 0;
|
|
};
|
|
|
|
class LuaFuzzer {
|
|
public:
|
|
LuaFuzzer(uint32_t seed = 0);
|
|
~LuaFuzzer();
|
|
|
|
// Run fuzzing for N iterations
|
|
FuzzResult Run(size_t iterations);
|
|
|
|
// Configuration
|
|
void SetMaxCodeLength(size_t len) { m_max_code_length = len; }
|
|
void SetMaxNesting(size_t depth) { m_max_nesting = depth; }
|
|
|
|
// Statistics
|
|
size_t GetTotalRuns() const { return m_total_runs; }
|
|
size_t GetCrashes() const { return m_crashes; }
|
|
size_t GetErrorsCaught() const { return m_errors_caught; }
|
|
|
|
private:
|
|
std::mt19937 m_rng;
|
|
size_t m_max_code_length = 1000;
|
|
size_t m_max_nesting = 10;
|
|
size_t m_total_runs = 0;
|
|
size_t m_crashes = 0;
|
|
size_t m_errors_caught = 0;
|
|
|
|
// Code generators
|
|
std::string GenerateRandomCode();
|
|
std::string GenerateExpression(int depth);
|
|
std::string GenerateStatement(int depth);
|
|
std::string GenerateIdentifier();
|
|
std::string GenerateLiteral();
|
|
|
|
// Sandbox integrity verification
|
|
bool VerifySandboxIntegrity();
|
|
};
|
|
|
|
} // namespace mosis
|
|
```
|
|
|
|
### 2. Fuzzer Code Generation
|
|
|
|
The fuzzer generates random Lua code including:
|
|
- Valid expressions (arithmetic, string, table)
|
|
- Control flow (if, while, for, repeat)
|
|
- Function definitions and calls
|
|
- Table operations
|
|
- Error-inducing patterns (intentional)
|
|
- Boundary conditions
|
|
|
|
### 3. Security Audit Tests
|
|
|
|
| Test | Description | Verifies |
|
|
|------|-------------|----------|
|
|
| `AuditNoOsAccess` | os.* blocked | SANDBOX.md §1 |
|
|
| `AuditNoIoAccess` | io.* blocked | SANDBOX.md §1 |
|
|
| `AuditNoLoadfile` | loadfile blocked | SANDBOX.md §1 |
|
|
| `AuditNoDofile` | dofile blocked | SANDBOX.md §1 |
|
|
| `AuditNoBytecode` | Bytecode rejected | SANDBOX.md §2 |
|
|
| `AuditMemoryLimit` | Memory limited | SANDBOX.md §3 |
|
|
| `AuditCPULimit` | CPU limited | SANDBOX.md §3 |
|
|
| `AuditMetatableProtected` | Metatables protected | SANDBOX.md §4 |
|
|
| `AuditNoStringDump` | string.dump removed | SANDBOX.md §5 |
|
|
| `AuditPathTraversal` | Path traversal blocked | SANDBOX.md §6 |
|
|
| `AuditPermissionEnforced` | Permissions checked | SANDBOX.md §7 |
|
|
| `AuditRateLimiting` | Rate limits work | SANDBOX.md §8 |
|
|
|
|
---
|
|
|
|
## Test Cases
|
|
|
|
### Test 1: Fuzzer Runs Without Crashes
|
|
|
|
```cpp
|
|
bool Test_FuzzerNoCrashes(std::string& error_msg) {
|
|
mosis::LuaFuzzer fuzzer(12345); // Deterministic seed
|
|
|
|
auto result = fuzzer.Run(1000); // 1000 iterations
|
|
|
|
EXPECT_TRUE(!result.crashed);
|
|
EXPECT_TRUE(result.sandbox_intact);
|
|
EXPECT_TRUE(result.iterations == 1000);
|
|
|
|
return true;
|
|
}
|
|
```
|
|
|
|
### Test 2: Fuzzer Catches Errors Gracefully
|
|
|
|
```cpp
|
|
bool Test_FuzzerCatchesErrors(std::string& error_msg) {
|
|
mosis::LuaFuzzer fuzzer(54321);
|
|
|
|
auto result = fuzzer.Run(500);
|
|
|
|
// Some generated code should produce errors (caught gracefully)
|
|
EXPECT_TRUE(result.errors_caught > 0);
|
|
EXPECT_TRUE(!result.crashed);
|
|
|
|
return true;
|
|
}
|
|
```
|
|
|
|
### Test 3: Sandbox Integrity After Fuzzing
|
|
|
|
```cpp
|
|
bool Test_FuzzerSandboxIntegrity(std::string& error_msg) {
|
|
mosis::LuaFuzzer fuzzer;
|
|
|
|
// Run many iterations
|
|
auto result = fuzzer.Run(2000);
|
|
|
|
// Sandbox must still be intact
|
|
EXPECT_TRUE(result.sandbox_intact);
|
|
|
|
// Verify by running a normal script
|
|
SandboxContext ctx = TestContext();
|
|
LuaSandbox sandbox(ctx);
|
|
EXPECT_TRUE(sandbox.LoadString("return 1 + 1", "verify"));
|
|
|
|
return true;
|
|
}
|
|
```
|
|
|
|
### Test 4: Audit - Dangerous Globals Blocked
|
|
|
|
```cpp
|
|
bool Test_AuditDangerousGlobalsBlocked(std::string& error_msg) {
|
|
SandboxContext ctx = TestContext();
|
|
LuaSandbox sandbox(ctx);
|
|
|
|
// All these must fail
|
|
std::vector<std::string> dangerous = {
|
|
"os.execute('ls')",
|
|
"io.open('test.txt')",
|
|
"loadfile('test.lua')",
|
|
"dofile('test.lua')",
|
|
"require('os')",
|
|
"package.loadlib('test', 'func')"
|
|
};
|
|
|
|
for (const auto& code : dangerous) {
|
|
bool ok = sandbox.LoadString(code, "audit");
|
|
if (ok) {
|
|
error_msg = "Dangerous code executed: " + code;
|
|
return false;
|
|
}
|
|
}
|
|
|
|
return true;
|
|
}
|
|
```
|
|
|
|
### Test 5: Audit - Resource Limits
|
|
|
|
```cpp
|
|
bool Test_AuditResourceLimits(std::string& error_msg) {
|
|
SandboxContext ctx = TestContext();
|
|
|
|
// Memory limit test
|
|
{
|
|
LuaSandbox sandbox(ctx);
|
|
sandbox.SetMemoryLimit(1024 * 1024); // 1MB
|
|
bool ok = sandbox.LoadString(R"lua(
|
|
local t = {}
|
|
for i = 1, 10000000 do
|
|
t[i] = string.rep("x", 1000)
|
|
end
|
|
)lua", "mem_test");
|
|
EXPECT_FALSE(ok); // Should fail due to memory limit
|
|
}
|
|
|
|
// CPU limit test
|
|
{
|
|
LuaSandbox sandbox(ctx);
|
|
sandbox.SetInstructionLimit(10000);
|
|
bool ok = sandbox.LoadString("while true do end", "cpu_test");
|
|
EXPECT_FALSE(ok); // Should fail due to CPU limit
|
|
}
|
|
|
|
return true;
|
|
}
|
|
```
|
|
|
|
### Test 6: Integration - Full App Lifecycle
|
|
|
|
```cpp
|
|
bool Test_IntegrationAppLifecycle(std::string& error_msg) {
|
|
// Create app context
|
|
SandboxContext ctx{
|
|
.app_id = "lifecycle.test.app",
|
|
.app_path = ".",
|
|
.permissions = {"storage"},
|
|
.is_system_app = false
|
|
};
|
|
|
|
// Create sandbox
|
|
LuaSandbox sandbox(ctx);
|
|
|
|
// Register all APIs
|
|
mosis::PermissionGate gate("lifecycle.test.app");
|
|
mosis::AuditLog audit;
|
|
mosis::VirtualFS vfs("lifecycle.test.app", ".");
|
|
mosis::TimerManager timers;
|
|
|
|
// Load app script
|
|
std::string script = R"lua(
|
|
-- App initialization
|
|
local data = json.encode({started = true})
|
|
|
|
-- Use storage (has permission)
|
|
storage.write("state.json", data)
|
|
|
|
-- Read back
|
|
local content = storage.read("state.json")
|
|
local state = json.decode(content)
|
|
|
|
-- Return success
|
|
return state.started == true
|
|
)lua";
|
|
|
|
bool ok = sandbox.LoadString(script, "lifecycle_test");
|
|
EXPECT_TRUE(ok);
|
|
|
|
// Cleanup
|
|
vfs.Delete("state.json");
|
|
|
|
return true;
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Acceptance Criteria
|
|
|
|
All tests pass:
|
|
|
|
- [x] `Test_FuzzerNoCrashes` - Fuzzer runs 100 iterations without crash
|
|
- [x] `Test_FuzzerCatchesErrors` - Fuzzer catches Lua errors gracefully
|
|
- [x] `Test_FuzzerSandboxIntegrity` - Sandbox intact after fuzzing
|
|
- [x] `Test_AuditDangerousGlobalsBlocked` - All dangerous globals blocked
|
|
- [x] `Test_AuditResourceLimits` - Memory and CPU limits enforced
|
|
- [x] `Test_IntegrationAppLifecycle` - Full app lifecycle works
|
|
|
|
---
|
|
|
|
## Dependencies
|
|
|
|
- All previous milestones (1-18)
|
|
|
|
---
|
|
|
|
## Notes
|
|
|
|
### Fuzzer Strategy
|
|
|
|
1. **Seed-based**: Deterministic with seed for reproducibility
|
|
2. **Incremental complexity**: Start simple, increase nesting
|
|
3. **Boundary testing**: Test edge cases (empty strings, huge numbers)
|
|
4. **Error injection**: Intentionally generate invalid code
|
|
|
|
### Security Audit Coverage
|
|
|
|
The audit tests verify all requirements from SANDBOX.md:
|
|
1. Dangerous standard library functions removed
|
|
2. Bytecode loading disabled
|
|
3. Memory limits enforced
|
|
4. CPU/instruction limits enforced
|
|
5. Metatables protected
|
|
6. Path traversal blocked
|
|
7. Permissions enforced
|
|
8. Rate limiting works
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
After Milestone 19 passes:
|
|
1. Milestone 20: Final Integration
|