148 Commits

Author SHA1 Message Date
Sybren A. Stüvel
183bb23d2a Tests: sleep test using bigger time steps
Since every mocked clock time step also waits for 1ms to give other
goroutines a chance to run, it took too much wallclock time to mock-sleep
for 47 seconds with 100ms increments.

Stepping the mocked clock with 1s increments makes the test 10x faster.
2022-03-04 12:35:01 +01:00
Sybren A. Stüvel
dbb9c71df8 Tests: more unified way to do database tests 2022-03-04 12:33:45 +01:00
Sybren A. Stüvel
f497ac8536 Cleanup: add and remove some comments 2022-03-04 12:19:19 +01:00
Sybren A. Stüvel
3bfd5a339f Fix database tests getting interrupted
The root cause was a 2nd `context.Context()` that was used in
`constructTestJob()`, which cancelled when that function returned.

The cancellation of the context caused an interrupt in the SQLite
driver, which got into a race condition and could cause an interrupt on
a subsequent database query.
2022-03-04 12:19:19 +01:00
Sybren A. Stüvel
de150567b0 Manager: avoid double error message 2022-03-04 11:37:29 +01:00
Sybren A. Stüvel
cd2fe8170e Errors: remove "error" prefix from message
Instead of returning an error "error doing X", just return "doing X". The
fact that it's returned as an error object says enough about that it's
an error.

This also makes it easier to chain error messages, without seeing the
word "error" in every part of the chain.
2022-03-04 11:30:31 +01:00
Sybren A. Stüvel
c91e7b1cac Manager: solve failing unittests by implementing some filepath functions
Both Go's standard `path` and `path/filepath` packages are too limiting to
work well for Flamenco. The former assumes Linux/POSIX paths, the latter
only works with platform-native paths. Neither can work with Windows paths
on Linux, or Linux paths on Windows.
2022-03-03 15:22:43 +01:00
Sybren A. Stüvel
b9609f8866 Cleanup: remove unused code 2022-03-03 13:52:57 +01:00
Sybren A. Stüvel
641ed7ace9 Manager: make Gorm use Zerolog for logging
A wrapper for Zerolog implements the Gorm logger interface. This gives
us coloured output on Windows, and uniform-looking logs in production.
2022-03-03 13:52:50 +01:00
Sybren A. Stüvel
8824489980 Manager: use in-memory SQLite database for testing
The on-disk database that was used before caused issues with tests running
in parallel. Not only is there the theoretical issue of tests seeing each
other's data (this didn't happen, but could), there was also the practical
issue of one test running while the other tried to erase the database file
(which fails on Windows due to file locking).
2022-03-03 13:51:55 +01:00
Sybren A. Stüvel
9b9c6bffff Replace self-hacked SQLite Gorm driver with 3rd party one
The new Gorm driver is made by the creators of the pure-Go SQLite library
we were already using.
2022-03-03 13:48:14 +01:00
Sybren A. Stüvel
2b04623e00 Manager: fix DB transaction isolation issue in task scheduler
The created transaction wasn't actually used for the should-be-in-the-
transaction queries. That's now resolved.
2022-03-03 13:46:27 +01:00
Sybren A. Stüvel
9643bf768e Manager: Fix DB migration error of not-null columns
Where the PostgreSQL DB migration code could handle `NOT NULL` columns just
fine, SQLite has less table-altering functionality. As a result, migrations
have to copy entire database tables, which doesn't play well with
not-nullable columns.
2022-03-03 12:10:13 +01:00
Sybren A. Stüvel
a7654394b4 Worker: update default task types to match what's currently in use
Flamenco v2 had a different classification of task types than Flamenco v3,
and the defaults still referenced the v2 classifications.
2022-03-03 11:51:53 +01:00
Sybren A. Stüvel
ddd0c7602b Change application name to "Flamenco 3"
"Flamenco NG PoC" was the name of the proof of concept, and it's no longer
in use.
2022-03-01 20:45:13 +01:00
Sybren A. Stüvel
47e36c927c Change package URL to the blender.org repository 2022-03-01 20:45:09 +01:00
Sybren A. Stüvel
e70a44a146 Manager: switch from PostgreSQL to SQLite
This includes a modified copy of the Gorm SQLite backend, adjusted to use
https://modernc.org/sqlite instead.
2022-03-01 18:50:31 +01:00
Sybren A. Stüvel
bbc6a3f69e Tweak some logging 2022-03-01 17:28:24 +01:00
Sybren A. Stüvel
0235ffcb4a Manager: avoid "no record found" error in task scheduler
It's fine when there is no task for a worker, so having Gorm log an error
was just causing noise.
2022-03-01 11:52:28 +01:00
Sybren A. Stüvel
7e5a631f33 Cleanup: refactor updateJobAfterTaskStatusChange()
Break up a complex function into smaller functions.
2022-02-28 12:50:34 +01:00
Sybren A. Stüvel
fab988295d Manager: remove task scheduler SQL debug logs 2022-02-28 12:07:23 +01:00
Sybren A. Stüvel
b5b7b228ed Worker: capture panics in 'active' state and cleanly sign off before dying 2022-02-28 12:07:10 +01:00
Sybren A. Stüvel
7689a988b1 Manager: re-queue tasks of worker when signing off 2022-02-28 12:06:50 +01:00
Sybren A. Stüvel
32af1ffaef Manager: actually pass context to Gorm queries 2022-02-28 11:53:31 +01:00
Sybren A. Stüvel
3d854078ba Manager: integrate task state machine into API implementation 2022-02-25 16:30:27 +01:00
Sybren A. Stüvel
17e622ebc3 Some build & filename tweaks to allow building from scratch again
The build chain got a bit confused when doing things from scratch, as
`test_support.go` was used in the non-test builds. Renaming it to
`support_test.go` was the easiest way to avoid that.
2022-02-25 16:17:49 +01:00
Sybren A. Stüvel
9a5bbb4131 Manager: implement persistence layer interface for task status machine
Implement the functions used by the task status machine in the DB
persistence layer.
2022-02-25 14:34:29 +01:00
Sybren A. Stüvel
7279f2e35f Manager: task state machine, handle job status -> task status changes
Direct copy of the Flamenco Server Python code, for handling the change
of a job's status to trigger status changes on its tasks.

Not yet connected to the rest of the Manager logic.
2022-02-25 12:30:40 +01:00
Sybren A. Stüvel
e8707171b4 Manager: add test for saving task status change with error on job save 2022-02-25 11:07:06 +01:00
Sybren A. Stüvel
d2f1cf3614 Cleanup: move test helper functions down in the file
This way it starts with the actual tests.
2022-02-25 11:02:47 +01:00
Sybren A. Stüvel
a3a0d97222 Manager: more task state change tests 2022-02-24 17:58:02 +01:00
Sybren A. Stüvel
df77d09aa6 Start of a task/job state machine
The task status change → job status change code is a direct port of the
Flamenco Server v2 code written in Python.

There is no job status change → task status changes logic yet, and the
tests are also far from complete.
2022-02-24 16:42:38 +01:00
Sybren A. Stüvel
64db518f15 Cleanup: remove unused code 2022-02-24 12:36:29 +01:00
Sybren A. Stüvel
7420177209 Manager: use api.JobStatus in persistence layer as well 2022-02-24 11:54:35 +01:00
Sybren A. Stüvel
7e776167bb Manager: use api.TaskStatus in persistence layer as well 2022-02-24 11:53:05 +01:00
Sybren A. Stüvel
2a1f5a0df4 Worker: buffer task updates when Manager offline
Queue task updates in an SQLite database when the Manager is
unreachable, periodically checking whether they can be flushed.
2022-02-22 19:05:33 +01:00
Sybren A. Stüvel
e948c5d936 Worker: avoid prematurely exiting 'awake' state 2022-02-22 19:01:01 +01:00
Sybren A. Stüvel
5c3a5124f3 Cleanup: add missing comment 2022-02-22 19:00:44 +01:00
Sybren A. Stüvel
b462e8db1a Worker: wrap api.ClientWithResponsesInterface in a local interface
This makes it possible to locally create a mock object for the API client.
2022-02-22 17:13:04 +01:00
Sybren A. Stüvel
a650aa4cbd Worker: Blender command, don't put current time in log lines
The time has to be put in at a different abstraction level.
2022-02-22 15:36:20 +01:00
Sybren A. Stüvel
adf7113b34 Worker: log chunker, also flush log after certain time
This flushes the log when the previous `Append()` call was too long ago.
Note that this doesn't flush after X seconds of silence; a call to
`Append()` or `Flush()` still has to happen in order to do the flushing.
2022-02-22 15:34:37 +01:00
Sybren A. Stüvel
45a95ca4c2 Worker: use time service to log current time 2022-02-22 15:33:30 +01:00
Sybren A. Stüvel
66186e460e Worker: upload logs in chunks to Manager 2022-02-22 15:17:35 +01:00
Sybren A. Stüvel
9a5047a94d Worker: move command_exe.go interface definitions 2022-02-22 14:42:57 +01:00
Sybren A. Stüvel
0525bd726c Worker: actually execute blender and log its output 2022-02-22 13:55:28 +01:00
Sybren A. Stüvel
7d67a1dc7a Manager: default paths for executables are now relative
This assumes that `blender` and `ffmpeg` are available on the commandline.
2022-02-22 13:55:03 +01:00
Sybren A. Stüvel
32c7a1fd84 Worker: Blender command, parse CLI arguments from 'exe' parameter 2022-02-22 12:52:16 +01:00
Sybren A. Stüvel
db4505e5fd Cleanup: move Blender command param parsing to its own function 2022-02-22 12:38:25 +01:00
Sybren A. Stüvel
20965691d0 Add test for Blender command and better command parameter parsing 2022-02-22 12:36:37 +01:00
Sybren A. Stüvel
e9a94eecae Worker: add CLI runner interface and move test/mock code around a bit 2022-02-22 12:35:43 +01:00