blender/flamenco
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
211 Commits 1 Branch 0 Tags
Commit Graph

50 Commits

Author SHA1 Message Date
Sybren A. Stüvel
b9609f8866 Cleanup: remove unused code 2022-03-03 13:52:57 +01:00
Sybren A. Stüvel
641ed7ace9 Manager: make Gorm use Zerolog for logging 
A wrapper for Zerolog implements the Gorm logger interface. This gives
us coloured output on Windows, and uniform-looking logs in production.
 2022-03-03 13:52:50 +01:00
Sybren A. Stüvel
8824489980 Manager: use in-memory SQLite database for testing 
The on-disk database that was used before caused issues with tests running
in parallel. Not only is there the theoretical issue of tests seeing each
other's data (this didn't happen, but could), there was also the practical
issue of one test running while the other tried to erase the database file
(which fails on Windows due to file locking).
 2022-03-03 13:51:55 +01:00
Sybren A. Stüvel
9b9c6bffff Replace self-hacked SQLite Gorm driver with 3rd party one 
The new Gorm driver is made by the creators of the pure-Go SQLite library
we were already using.
 2022-03-03 13:48:14 +01:00
Sybren A. Stüvel
2b04623e00 Manager: fix DB transaction isolation issue in task scheduler 
The created transaction wasn't actually used for the should-be-in-the-
transaction queries. That's now resolved.
 2022-03-03 13:46:27 +01:00
Sybren A. Stüvel
9643bf768e Manager: Fix DB migration error of not-null columns 
Where the PostgreSQL DB migration code could handle `NOT NULL` columns just
fine, SQLite has less table-altering functionality. As a result, migrations
have to copy entire database tables, which doesn't play well with
not-nullable columns.
 2022-03-03 12:10:13 +01:00
Sybren A. Stüvel
47e36c927c Change package URL to the blender.org repository 2022-03-01 20:45:09 +01:00
Sybren A. Stüvel
e70a44a146 Manager: switch from PostgreSQL to SQLite 
This includes a modified copy of the Gorm SQLite backend, adjusted to use
https://modernc.org/sqlite instead.
 2022-03-01 18:50:31 +01:00
Sybren A. Stüvel
0235ffcb4a Manager: avoid "no record found" error in task scheduler 
It's fine when there is no task for a worker, so having Gorm log an error
was just causing noise.
 2022-03-01 11:52:28 +01:00
Sybren A. Stüvel
fab988295d Manager: remove task scheduler SQL debug logs 2022-02-28 12:07:23 +01:00
Sybren A. Stüvel
7689a988b1 Manager: re-queue tasks of worker when signing off 2022-02-28 12:06:50 +01:00
Sybren A. Stüvel
32af1ffaef Manager: actually pass context to Gorm queries 2022-02-28 11:53:31 +01:00
Sybren A. Stüvel
3d854078ba Manager: integrate task state machine into API implementation 2022-02-25 16:30:27 +01:00
Sybren A. Stüvel
9a5bbb4131 Manager: implement persistence layer interface for task status machine 
Implement the functions used by the task status machine in the DB
persistence layer.
 2022-02-25 14:34:29 +01:00
Sybren A. Stüvel
7420177209 Manager: use api.JobStatus in persistence layer as well 2022-02-24 11:54:35 +01:00
Sybren A. Stüvel
7e776167bb Manager: use api.TaskStatus in persistence layer as well 2022-02-24 11:53:05 +01:00
Sybren A. Stüvel
80df8fa6e4 DB Initialisation: try named parameters 
Should be tested on Windows, as that's where this code will be used most
often. As of now, untested.
 2022-02-22 12:06:54 +01:00
Sybren A. Stüvel
90a2140b8c Manager: store task logs to disk 2022-02-21 19:47:07 +01:00
Sybren A. Stüvel
b1b73de4ee Manager: Actually load config, and use in a few places 2022-02-21 19:02:07 +01:00
Sybren A. Stüvel
66ae9b3a64 Job persistence test: create authored job with explicit status 2022-02-21 18:04:52 +01:00
Sybren A. Stüvel
ef2bbd2845 Unified Command field names 
Some parts of Flamenco had a Command consist of "name + settings", and
other parts used "type + parameters" (with the same semantics). This is
now unified to "name + parameters".
 2022-02-21 18:03:51 +01:00
Sybren A. Stüvel
6bd1a86337 Start of DB initialisation functions 
Not properly working yet
 2022-02-21 15:25:56 +01:00
Sybren A. Stüvel
399c8af750 Correctly handle workers assigned to tasks + simple task updates 2022-02-17 17:30:52 +01:00
Sybren A. Stüvel
270c54fdb7 More status change acks & checks to get stable flow between worker states 2022-02-15 17:46:37 +01:00
Sybren A. Stüvel
c4df62d5d4 Start of sending task updates to Manager 
This includes a mocking framework for unittests.
 2022-02-15 15:58:24 +01:00
Sybren A. Stüvel
9543d6221b Wrap error with %w 2022-02-15 15:56:54 +01:00
Sybren A. Stüvel
50088b4c94 Save worker info on sign-on (not just on registration) 2022-02-15 10:57:29 +01:00
Sybren A. Stüvel
0457809641 Scheduler: filter on supported task types & assigned worker ID 2022-02-14 18:00:43 +01:00
Sybren A. Stüvel
4aafb782ac Scheduler: Assign task to worker 2022-02-14 17:47:26 +01:00
Sybren A. Stüvel
eea219c3e2 Move task scheduler into persistence layer 2022-02-14 16:53:21 +01:00
Sybren A. Stüvel
8e01c069d1 DB: task dependencies should be cascade-deleted with their tasks 2022-02-14 15:07:14 +01:00
Sybren A. Stüvel
97ab93d996 Initial task scheduler implementation 2022-02-01 17:17:19 +01:00
Sybren A. Stüvel
fad2dc3042 Job authoring: handling of task dependencies + some bugfixes 2022-02-01 17:17:19 +01:00
Sybren A. Stüvel
e598397ba4 Persistence: move some DB to API struct conversion to API implementation 2022-02-01 16:22:10 +01:00
Sybren A. Stüvel
862ed96af7 Make persistence.CreateTestDB public so it can be used from other packages 2022-02-01 10:57:33 +01:00
Sybren A. Stüvel
2ca8858c28 Only update status field in DB when worker changes status 2022-02-01 10:16:10 +01:00
Sybren A. Stüvel
be89349632 Very basic non-functional framework for a task runner 
Also has some login/logout functionality for storing stuff in the DB.
 2022-01-31 16:05:27 +01:00
Sybren A. Stüvel
c501899185 Ported lots of stuff from gitlab.com/dr.sybren/flamenco-worker-go 
Much isn't working though.
 2022-01-28 17:02:50 +01:00
Sybren A. Stüvel
28a56f3d91 Store workers in database when registering 2022-01-28 15:31:39 +01:00
Sybren A. Stüvel
931fd1a24c Move db/jobs tests to persistence/jobs.go 2022-01-28 14:53:02 +01:00
Sybren A. Stüvel
3b1cb3c187 Remove sqlite library & run go mod tidy 2022-01-28 14:52:46 +01:00
Sybren A. Stüvel
0b3311b0a7 Change parameter format of blender-render command 2022-01-28 14:48:19 +01:00
Sybren A. Stüvel
30518ca3af Store tasks & commands in the database 2022-01-25 18:25:26 +01:00
Sybren A. Stüvel
edda8f21cf Make tests work on PostgreSQL 2022-01-25 18:12:01 +01:00
Sybren A. Stüvel
0f8bacd2e5 Store compiled jobs with GORM in PostgreSQL 2022-01-24 18:07:12 +01:00
Sybren A. Stüvel
b446b02739 Change package URL 
Change package URL from `gitlab.com/blender/flamenco-goja-test` to
`gitlab.com/blender/flamenco-ng-poc` so that it matches the actual URL
on Gitlab.
 2022-01-24 14:28:00 +01:00
Sybren A. Stüvel
0629728ce9 Add frame chunker and make unit test for simple blender render succeed 2022-01-13 16:32:08 +01:00
Sybren A. Stüvel
54933d0262 API: add Fetch Job endpoint 
This is made with plain sqlite. It's probably a good idea to move to
something like GORM instead.
 2022-01-11 18:38:14 +01:00
Sybren A. Stüvel
f0ae7d33e9 Connect API, job compiler, and persistence layer 2022-01-11 18:07:34 +01:00
Sybren A. Stüvel
0c9c99806b Start of database interface 
Contains a dummy migration because the migrate package will error out if
it cannot find any migration files.
 2022-01-11 18:07:34 +01:00