1131 Commits

Author SHA1 Message Date
Sybren A. Stüvel
b04889dd79 Upgraded all Go dependencies
Ran `go get -t -u ./...` to get all the main + test dependencies upgraded.
2022-07-18 19:39:43 +02:00
Sybren A. Stüvel
42b12a7dc7 cleanup: go mod tidy
Just the effect of running `go mod tidy`.
2022-07-18 19:37:31 +02:00
Sybren A. Stüvel
0a5f87bc5a Sleep Scheduler: perform first check at startup
Instead of waiting for a minute, run the first sleep scheduler iteration
at startup.
2022-07-18 19:30:38 +02:00
Sybren A. Stüvel
83467e4c60 Sleep schedule: store 'next check' timestamp in UTC
SQLite doesn't parse the timezone info, so timestamps should always be in
UTC.
2022-07-18 19:30:17 +02:00
Sybren A. Stüvel
3baac0a2d8 Manager: reduce log level when worker asks task but has wrong status
This can happen quite often and it's fine, so it's not worth a warning.
2022-07-18 19:26:49 +02:00
Sybren A. Stüvel
eb65ac9085 Manager: add source location when logging in debug/trace mode
Just a little debugging tool.
2022-07-18 19:05:19 +02:00
Sybren A. Stüvel
24f921b0c8 Manager: add more logging when worker cannot be marked as 'seen'
SQLite often errors out on this with only `interrupted (9)` as message.
This logging should at least tell us whether it's our own "background
context" timing out, or whether something else fishy is going on.
2022-07-18 19:04:15 +02:00
Sybren A. Stüvel
bfd6746f78 Manager: consult the sleep schedule on worker sign-on
If there is no status change queued for the Worker, the sleep schedule
should determine its initial status.
2022-07-18 18:25:24 +02:00
Sybren A. Stüvel
bc725ea7dc Manager: mark worker as 'seen' when calling the WorkerState operation
Fix workers timing out when they're `asleep`. When sleeping, the Worker
will call the `WorkerState` operation to see if they have to wake up, but
that didn't mark the workers as "seen". As a result, a sleeping worker
would always time out.
2022-07-18 17:56:56 +02:00
Sybren A. Stüvel
63db0dc75c OAPI: regenerate code 2022-07-18 17:52:13 +02:00
Sybren A. Stüvel
f60849189a OAPI: Clean up FlamencoVersion schema definition
Change the `FlamencoVersion` schema definition so that it follows the style
of the other schema definitions:

- List properties before mentioning which are required.
- Put quotes around the property names, so that they stand out from the
  other YAML keys.
2022-07-18 17:47:57 +02:00
Sybren A. Stüvel
f16012d246 Static site: add documentation about calling the Flamenco API 2022-07-18 17:46:37 +02:00
Sybren A. Stüvel
47e517a3a5 Worker: cleanly sign off after flushing buffer
When running the Worker with the `-flush` CLI argument, actually sign off
from the Manager before shutting down.
2022-07-18 16:36:45 +02:00
Sybren A. Stüvel
0697f71b62 Manager: run some operations in a background context
Run some API operations in a background context. This should prevent some
of the SQLite "interrupted" errors, as those can occur when the context
closes while a query is running.

The API operations that Workers use are now mostly running in a separate
background context, at least from the moment onward when they can run
independently of the Worker connection.
2022-07-18 16:26:06 +02:00
Sybren A. Stüvel
778ad6927b Manager: nicer logging of its own URLs
Log the URLs in an easier to read and also easier-to-copy-paste format.
2022-07-18 15:37:07 +02:00
Sybren A. Stüvel
43e8f3f623 Manager: improve the "my own URLs" construction
Improve the "my own URLs" construction, such that:
- IPv6 link-local addresses are always skipped. They require a "zone index"
  string, typically the interface name, so something like
  `[fe80::cafe:f00d%eth0]`. This is not supported by web browsers, so the
  URLs would be of limited use. Furthermore, they require the interface
  name of the side initiating the connection, whereas this code is used to
  answer the question "how can this machine be reached as a server?"
- IPv4 addresses are sorted before IPv6 addresses. Even though I like IPv6
  a lot, IPv4 is still more familiar to people.
- Loopback addresses (::1, 127.0.0.1) are sorted last, so that the First-
  Time Wizard is most likely to use the bigger-scoped address.
2022-07-18 15:36:43 +02:00
Sybren A. Stüvel
bf5bf86f03 Manager: log own URLs at startup
Log the URLs at which the Manager thinks it should be reachable, at
startup.
2022-07-18 14:45:49 +02:00
Sybren A. Stüvel
e91623557a Worker: log which URLs were tried when auto-discovery failed
When the Worker cannot find any Manager, log which URLs were tried.
2022-07-18 14:14:02 +02:00
Sybren A. Stüvel
ad57070a2d Manager: reduce log level of "loading configuration" message
Every time the web interface starts, it queries the config to see whether
it should be in first-time-wizard mode or not. This caused unnecessary
info-level logging.

In the future it would be better to load the config file just once,
instead.
2022-07-18 14:11:22 +02:00
Sybren A. Stüvel
658a3d7a85 Worker Timeout: subject all but offline/error workers to timeout checks
Workers that are in `starting`, `asleep`, or `testing` state should also
be subject to the timeout check, not just workers in `awake` state.
2022-07-18 11:30:39 +02:00
Sybren A. Stüvel
a6ca3f7bdc Sleep Scheduler: reduce check interval and log level
Reduce the check interval and the log level of "nothing to do" messages,
from "developer friendly" to "actually useful".
2022-07-17 17:31:51 +02:00
Sybren A. Stüvel
d7b164133a Sleep Scheduler implementation for the Manager
The Manager now has a sleep scheduler for Workers. The API and background
service work, but there is no web interface yet.

Manifest Task: T99397
2022-07-17 17:27:32 +02:00
Sybren A. Stüvel
3133bd2487 OAPI: regenerate code 2022-07-17 17:25:53 +02:00
Sybren A. Stüvel
f23ba6d53e OAPI: correction to the sleep schedule example
The `days_of_week` field must be space-separated, not comma-separated.
2022-07-17 17:25:05 +02:00
Sybren A. Stüvel
627996525e Manager: implement operations for getting & setting worker sleep schedule
This is just the API, no web interface yet.

Manifest Task: T99397
2022-07-16 16:00:25 +02:00
Sybren A. Stüvel
0e92004f2a OAPI: regenerate code 2022-07-16 15:59:48 +02:00
Sybren A. Stüvel
26f92503cf OAPI: operations for getting & setting worker sleep schedule
Manifest Task: T99397
2022-07-16 15:57:41 +02:00
Sybren A. Stüvel
120db29351 Web: show "Last seen: never" when worker has never been seen
Show "Last Seen: never" in the Worker details, if the Worker has never
been seen.
2022-07-16 15:35:07 +02:00
Sybren A. Stüvel
32f4ceedf3 Web: use backendURL() function to construct API and Add-on URLs
Use `backendURL()` function to construct API and Add-on URLs, so that they
work regardless of what is serving the webapp (Flamenco Manager or the
Vite development server).
2022-07-16 15:34:32 +02:00
Sybren A. Stüvel
7d049d3cda Manager: make artificial API delay actually optional
For development of the web interface, to get a less predictable order of
asynchronous requests, the API responses were artificially delayed. This
was supposed to be optional, to be enabled via the `-delay` CLI argument,
but somehow the optionalness either never made it in or was mysteriously
removed.
2022-07-16 13:24:59 +02:00
Sybren A. Stüvel
b3701ef493 Web: style task log buttons
Style the task log buttons like the other buttons.
2022-07-16 13:23:06 +02:00
Sybren A. Stüvel
a97913bebb Web: add "Follow Task Log" button
The "Follow Task Log" button should make it easier to discover the task
log viewer in the footer pop-over.
2022-07-16 13:22:11 +02:00
Sybren A. Stüvel
726129446d T99730: Allow access to full task log
The web interface has a button that opens the task log in a new window.
This might need some restyling ;-)
2022-07-16 12:55:41 +02:00
Sybren A. Stüvel
e4627daf4b OAPI: regenerate code 2022-07-16 12:54:11 +02:00
Sybren A. Stüvel
d170cda3ae OAPI: change task log fetching from providing the log to providing metadata
The task log API endpoint was loading the entire log into RAM, then sending
it as response. This makes display in a browser also a bit harder.

The API endpoint now returns some JSON with info about the task log,
including its size and which URL can be used to download it.

Manifest task: T99730
2022-07-16 12:52:23 +02:00
Sybren A. Stüvel
f19d800e35 Web: fix task log loading
Not sure how this ever worked before. It did though ;-)
2022-07-16 11:28:45 +02:00
Sybren A. Stüvel
686295090b Manager: implement endpoint for getting the full task log
Previously only the log tail was available, which is fine for many cases,
but for serious debugging the entire log is needed.

Manifest task: T99730
2022-07-16 11:13:31 +02:00
Sybren A. Stüvel
fee0717179 Makefile: on Windows, run unix2dos on generated files
The generators always produce UNIX line-ends. This creates false file
modifications with Git on Windows. Convert them to DOS line-ends to avoid
this.

This was already done for the Go code; this commit just copies the approach
to the Python and JavaScript generators.
2022-07-16 11:12:41 +02:00
Sybren A. Stüvel
e2434b44f2 OAPI: regenerate code 2022-07-16 11:11:34 +02:00
Sybren A. Stüvel
7521bff1a7 OAPI: add operation to read the entire task log
Manifest task: T99730
2022-07-16 11:05:58 +02:00
Sybren A. Stüvel
9cebcc9f7c Fix T99391: get rid of the last occasional scrollbar in the middle column
My way to get things working (I wouldn't call this a "solution" as I don't
know the root cause) was to emit two consecutive "reshuffled" events when
changing tabs in the job details, and to recalculate the table height when
the job type (so not the job itself, but its type info) is loaded.
2022-07-16 10:51:43 +02:00
Sybren A. Stüvel
ca586bf3fe Windows: Skip "inaccessible path" test
For some reason, on Windows, creating a directory with zero permissions
still allows creating a file in there. Just skip that part of the test.

The Explorer's properties panel of the directory also shows "Read Only
(only applies to files)", so at least that seems consistent.
2022-07-16 10:31:35 +02:00
Sybren A. Stüvel
859a261b05 Manager: on deletion of a worker, do not cascade to deletion of its tasks
Fix an issue where deleting a Worker would also delete the tasks it was
assigned to.
2022-07-15 17:00:25 +02:00
Sybren A. Stüvel
b511fad968 Manager: add profiler support
Add a `-pprof` CLI option to enable the profiler. It will expose profiler
info on the web interface at `/debug/pprof/`.

To have a nice view of this, including flame graphs, run:

```
go tool pprof -http localhost:8082 http://localhost:8080/debug/pprof/profile
```
2022-07-15 15:08:00 +02:00
Sybren A. Stüvel
904b6c0d73 Stresser: stress the Manager by querying for tasks to execute 2022-07-15 15:08:00 +02:00
Sybren A. Stüvel
26f07d85fd Gitignore stresser executable 2022-07-15 15:08:00 +02:00
Sybren A. Stüvel
1fceae3604 Manager: more efficient database queries
Be more selective in what's saved to the database to speed some things up.
Most importantly, this avoids saving the entire job when a task status is
updated or a task is assigned.
2022-07-15 15:08:00 +02:00
Sybren A. Stüvel
be77403114 Makefile: add flamenco-manager-without-webapp target
This builds Flamenco Manager, without rebuilding the webapp, and without
the race condition checker.
2022-07-15 15:08:00 +02:00
Sybren A. Stüvel
809036898b Stresser: cleanly sign off when shutting down 2022-07-15 15:08:00 +02:00
Sybren A. Stüvel
1055aabee2 Manager: optimise db.SaveActivity() query
Use an explicit `Select()` GORM call to avoid saving related objects.
2022-07-15 15:08:00 +02:00