276 Commits

Author SHA1 Message Date
Sybren A. Stüvel
18891dda91 Manager: implement FetchTask OAPI endpoint 2022-05-03 13:04:28 +02:00
Sybren A. Stüvel
4da7f67105 OAPI: generate code 2022-05-03 13:03:59 +02:00
Sybren A. Stüvel
891e791853 Manager: reduce log level of socketIO subscription changes 2022-05-03 12:04:27 +02:00
Sybren A. Stüvel
50c8cd39f2 Task update notifications via SocketIO
Manager now sends out task updates via SocketIO, and the web interface
handles those.

Note that there is a `BroadcastTaskUpdate()` function, but not a
`BroadcastNewTask`. The 'new job' broadcast is sent after the job's
tasks have been created, and thus there is no need for a separate
broadcast per task.
2022-05-03 11:26:24 +02:00
Sybren A. Stüvel
bb68488c5e Cleanup: Manager, add bit of documentation 2022-05-03 10:39:44 +02:00
Sybren A. Stüvel
9b330280b7 Add SocketIO subscription system for job-related updates
SocketIO clients can now send a message with `/subscription` event type
in order to subscribe to or unsubscribe from job-related updates.

These job-related updates themselves aren't sent yet, so this is a change
that's impossible to really test. The socketIO code for joining/leaving
rooms is called, though.
2022-05-02 18:36:14 +02:00
Sybren A. Stüvel
8d69bfe069 Cleanup: Manager, reorganise the socketio code a bit 2022-04-29 16:58:48 +02:00
Sybren A. Stüvel
629c073ed7 Manager: fix query for job tasks 2022-04-29 12:26:53 +02:00
Sybren A. Stüvel
992fc38604 OAPI: add endpoint for fetching the tasks of a job
Add `fetchJobTasks` operation to the Jobs API. This returns a summary of
each of the job's tasks, suitable for display in a task list view.

The actually used fields may need tweaking once we actually have a task
list view, but at least the functionality is there.
2022-04-22 12:52:57 +02:00
Sybren A. Stüvel
e399b14e66 Manager: cleanup, rename jobId to jobID
No functional changes.
2022-04-22 12:16:11 +02:00
Sybren A. Stüvel
0cd478a409 Manager: move FetchJob function into jobs_query.go
I want to put more of the "get stuff" code into `jobs_query.go`, keeping
`jobs.go` for creationg & manipulation.
2022-04-22 11:51:02 +02:00
Sybren A. Stüvel
e34a0ba6ea Worker: more granular locking when flushing upstream buffer
Only lock the database mutex when actual queries are performed, but not
during the entire flush loop.
2022-04-21 19:19:01 +02:00
Sybren A. Stüvel
8937a6f06f Cleanup: worker, remove debug timers
Remove accidentally committed debug timing code.
2022-04-21 19:14:09 +02:00
Sybren A. Stüvel
bcbacf6c42 Worker: fix race condition getting logger with worker status 2022-04-21 19:12:53 +02:00
Sybren A. Stüvel
6bdc198301 Manager: more graceful errors when receiving task update of unknown task
Return a 404 Not Found when the task can't be found, and a 500 on other
errors.
2022-04-21 19:06:18 +02:00
Sybren A. Stüvel
1176d85496 Worker: add -flush CLI option to just flush the buffer and quit
Add `-flush` CLI option to just flush the upstream queue and then quit.
2022-04-21 19:06:18 +02:00
Sybren A. Stüvel
90be370095 Manager: reduce password strength of Workers
The password check of worker API calls was 2 orders of magnitude slower
than actually handling the API call itself. Since the Worker authentication
is not that important (it's all on the same network anyway, and Worker
account registration is automatic too), lowering the BCrypt cost to the
minimum helps.

On my machine, this reduces the time for password checks from 50 to 2 ms.
2022-04-21 19:06:18 +02:00
Sybren A. Stüvel
65427ee38e Manager: use e.NoContent(http.StatusNoContent) to return "no content"
No functional changes, just the right call for the job.
2022-04-21 19:06:18 +02:00
Sybren A. Stüvel
79bac3a5f3 Manager: fix race condition in logging of worker properties
Dereferencing the `w *persistence.Worker` pointer should happen directly
in the function call, not in the zerolog callback function.
2022-04-21 19:06:18 +02:00
Sybren A. Stüvel
5466f65225 OAPI: add setJobStatus operation
Add API endpoint for updating the job status.
2022-04-21 19:06:18 +02:00
Sybren A. Stüvel
b699647ed4 OpenAPI: add activity field to Job schema 2022-04-21 12:40:25 +02:00
Sybren A. Stüvel
d79fde17f3 Manager: keep track of the reason of job status changes
To prepare for job status changes being requestable from the API, store
the reason for any status change on the job itself.

Not yet part of the API, just on the persistence layer.
2022-04-21 12:32:07 +02:00
Sybren A. Stüvel
954af37fd5 Manager: rename assertXXXResponse to assertResponseXXX
Rename test functions like `assertJSONResponse` to `assertResponseJSON`,
so that they get ordered together by autocompletion.

No functional changes.
2022-04-21 12:01:46 +02:00
Sybren A. Stüvel
c3b694ab2a Manager: wrap job/task errors in persistence layer
Avoid users of the persistence layer to have to test against Gorm errors,
by wrapping job/task errors in a new `PersistenceError` struct.

Instead of testing for `gorm.ErrRecordNotFound`, code can now test for
`persistence.ErrJobNotFound` or `persistence.ErrTaskNotFound`.
2022-04-21 11:54:59 +02:00
Sybren A. Stüvel
d099a31531 OAPI: add endpoint for getting a single job type
This will be used by the web frontend to determine which job settings
to show and which to hide.
2022-04-15 16:21:48 +02:00
Sybren A. Stüvel
d30befa2d7 Manager: add assert function for testing JSON responses
This makes it much easier to test an API response actually matches the
expected JSON values.
2022-04-15 16:14:17 +02:00
Sybren A. Stüvel
de3c4af8cb Manager: fix broken unit test 2022-04-15 14:37:41 +02:00
Sybren A. Stüvel
13e3607571 Manager: reduce logging of config loading
The logging was very verbose, and even though it was only at trace level,
a failing unit test would show them all.
2022-04-15 14:34:30 +02:00
Sybren A. Stüvel
e672289a11 OAPI: include all info for the jobs table in the JobUpdate schema
By having all info for the jobs table in the `JobUpdate` schema, it won't
have to query for the full job when a new job is added. This fetching of
the full job is now delayed until someone clicks on the table row.
2022-04-14 09:41:04 +02:00
Sybren A. Stüvel
555c935790 Web: Replace Vue 2 with Vue 3 webapp
Replace the Vue v2 webapp with a Vue v3 one, and embed the OpenAPI
client in the webapp itself (instead of being its own npm project).

- Vue v2.x -> v3.x
- Tabulator v4.x -> v5.1
- Moment JS -> replaced with Luxon JS
- Vue CLI/UI -> replaced with Vite
2022-04-12 12:34:49 +02:00
Sybren A. Stüvel
48417f7f14 Manager: Fix unittests after recent changes to the job compiler script
d98dbaa3 introduced a change to the job compiler, which wasn't taken into
account in the unit test.
2022-04-11 14:08:40 +02:00
Sybren A. Stüvel
c79983bafb Worker: include -r {fps} in FFmpeg command
This should force the output created by FFmpeg to match the desired FPS.
2022-04-11 12:09:28 +02:00
Sybren A. Stüvel
e9212de196 Worker: simplify "context done" checks 2022-04-09 16:57:39 +02:00
Sybren A. Stüvel
d98dbaa333 Worker: implement ffmpeg for frame-to-video conversion on Windows 2022-04-09 16:20:29 +02:00
Sybren A. Stüvel
38aedfe8cc Worker: make blender-finder find blender.exe and not blender-launcher.exe
`blender-launcher.exe` doesn't pipe Blender's stdout/stderr output, so it's
not useful for Flamenco.
2022-04-09 16:16:19 +02:00
Sybren A. Stüvel
0959179225 Worker: redirect stderr to stdout at the right moment
It should be done after stdout has been assigned a pipe, and not before.
2022-04-09 16:14:44 +02:00
Sybren A. Stüvel
4181708709 Worker: automatically find Blender on Windows via file association
On Windows, when the Manager doesn't explicitly point at a Blender to use,
use the the application associated with `.blend` files instead.
2022-04-09 15:05:34 +02:00
Sybren A. Stüvel
bf47afc32b Worker: reduce log level of "fetching tasks" message 2022-04-09 12:52:28 +02:00
Sybren A. Stüvel
7a19e2f38d Manager: use absolute storage path
This helps to get things consistent on Windows and Linux. Otherwise a path
like `/some/path` is absolute on one platform but not on the other. This is
mostly for getting the unit tests in this package to work on Windows, but
using absolute paths also helps in clarity of error logging.
2022-04-09 12:03:11 +02:00
Sybren A. Stüvel
cc20b850ec Worker: separate contexts of upstream buffer
The upstream buffer takes care of two things: communication with Flamenco
Manager (first context) and buffering things in a SQLite database (second
context). This commit separates those two contexts, so that shutting down
the application isn't going to prevent buffering things in the database.
2022-04-08 16:49:53 +02:00
Sybren A. Stüvel
e7fc2c6f6e Worker: add various extra error checks 2022-04-08 14:47:20 +02:00
Sybren A. Stüvel
1960b668aa Cleanup: remove unused code 2022-04-08 14:47:07 +02:00
Sybren A. Stüvel
2e2205c00e Manager: return error from sendAPIError()
Small bugfix.
2022-04-08 14:46:36 +02:00
Sybren A. Stüvel
62ea7dcf1d Worker: avoid task errors on shutdown
When shutting down, the main worker context closes. This causes the
subprocess to be killed, which in turn caused a task execution error. This
now no longer happens, as such errors are expected on shutdown and do not
indicate task failure.
2022-04-08 14:32:25 +02:00
Sybren A. Stüvel
06bf3c0482 Cleanup: manager, fix copy-paste from original OpenAPI example code 2022-04-08 12:04:58 +02:00
Sybren A. Stüvel
930d7497d7 OAPI: Better 'SQLITE_BUSY' error handling
SQLite can return `SQLITE_BUSY` errors when it's doing too many things at
the same time. This is now improved a bit by setting a 5-second timeout,
during which the SQLite driver will wait for the database to become
available. If that doesn't happen, Flamenco Manager will return a
`503 Service Unavailable` response so that the client knows to back off
a little.
2022-04-08 12:02:30 +02:00
Sybren A. Stüvel
89e130c04f Manager: update tests for inclusion of job name in job updates 2022-04-08 11:59:30 +02:00
Sybren A. Stüvel
df3f7b44b9 Hook up web interface to job updates 2022-04-07 18:46:09 +02:00
Sybren A. Stüvel
a476f39365 Manager: improve SocketIO event handling & logging 2022-04-05 16:34:32 +02:00
Sybren A. Stüvel
fdab07d987 Worker: slight clarification of error message 2022-04-05 16:28:22 +02:00