Orchestration Service
Testing new changes
Overview:
The Solsta Orchestration Service allows studios to view and manage concurrent installs or updates within a shared network to prevent bottleneck scenarios that constrain network performance.
Limiting concurrent downloads within a shared network can avoid performance degradation when there are many Solsta users or devices in a studio. In a throttled network, Solsta clients will check with the Orchestration Service endpoint prior to an install or update to ensure they are not oversaturating the network. The Orchestration Service will either allow the download or put it in a queue.
Visibility into All Downloads within the Organization
Beginning in version 1.13 of the Solsta Desktop Application, Organization Administrators are able to see ALL active or pending downloads within the organization, regardless of whether a network is throttled or not.
How Queueing Works:
When a network is throttled, the orchestration service will allow the number of concurrent downloads up to the set threshold for the network. All subsequent downloads beyond the threshold will be queued in a “pending” state. Clients will check in with the service every 15 seconds and automatically start their installs or updates as soon as their downloads are allowed. Additional queuing information:
- Threshold rules are set per network by specifying using CIDR network notation (e.g. 192.168.8.0/24)
- Device thresholds can also be set to limit the concurrent downloads on each device. We recommend setting this between 2 and 4, although machine resources and build composition will determine how many concurrent downloads each machine will be able to process in a timely manner. Note that more is not always better, as each concurrent download will compete for local bandwidth, memory and disk read/write resources. That is: it may be faster to process 10 downloads two at a time instead of 10 simultaneously.
- Threshold rules apply to ALL downloads in the network, including concurrent downloads on a single Solsta client. See note below on how device and network thresholds work together.
- Solsta clients that have no network threshold rules will not be impacted, however, Organization Administrators can still view all the active downloads in their organization.
- Paused downloads will leave the queue, resumed downloads will join the queue as new downloads
- Closing the Solsta client will gracefully remove any active downloads from the queue
- A crash or unexpected process termination will cause both active and pending downloads to be abandoned. In this scenario:
- Abandoned active downloads are cleared from the queue within 10 minutes if the client has not checked back in
- Abandoned pending downloads are cleared after 24 hours if the client has not checked back in. Note that these pending downloads will not hold up other downloads in the queue.
- A client recovering from an abandoned downloads will clear the abandoned downloads from the queue when it starts new session. The client can re-join the queue by restarting the previously abandoned downloads.
Network and Device Thresholds
When both device and network thresholds are set, the lower value will supersede the higher value (this will typically be the lower device threshold). For example, if a network rule with device threshold set to 2 and network threshold set to 4 is active:
- Machine A starts 3 concurrent downloads. 2 will be allowed per device limits, the 3rd will be queued.
- Two open download slots remain available for other machines in the network
Example Scenario
A studio’s network threshold is set to 5. A new build is deployed so 10 Solsta users receive an update notification. 8 of the users click “update” at roughly the same time.
- The first 5 users begin downloading the update immediately
- The next 3 users see a “Queued with organization…” status message on their Solsta client
- When the first of the original 5 active downloads completes, one of the 3 pending downloads begins within ~15 seconds
- The process repeats until all downloads complete and the queue is cleared
Manage Network Rules
Throttling downloads for a network can significantly impact a studio’s performance when downloading updates, therefore, only Organization-level Administrators can view and manage network rules.
To view and manage the network rules, log in as an Organization Administrator, then
- Click on the Organization tab
- Click on the Network Rules sub-tab
Create Network Rule
- Click on the Create Network Rule button on the right side of the UI. This brings up the Create Network Rule modal window.
- Source field: Enter an IP block using CIDR network notation (e.g. 192.168.8.0/24)
- Network Threshold field: Enter a positive integer for the maximum number of concurrent downloads to allow on this network
- Device Threshold field:Enter a positive integer for the maximum number of concurrent downloads to allow per device on this network
- Click Ok
Update Network Rule
- From the Network Rules sub-tab, find the rule you want to edit and click the Edit icon from the Actions column. This brings up the Edit Network Rule modal window.
- Source field: Enter an IP block using CIDR network notation (e.g. 192.168.8.0/24)
- Network Threshold field: Enter a positive integer for the maximum number of concurrent downloads to allow on this network
- Device Threshold field:Enter a positive integer for the maximum number of concurrent downloads to allow per device on this network
- Click Ok
Rule Propagation Time
Heed the warning in the UI: Changes may take up to 10 minutes to propagate.
Delete Network Rule
- From the Network Rules sub-tab, find the rule you want to delete and click the Trash icon from the Actions column. This brings up the Delete Network Rule modal window.
- Click Yes to confirm you want to delete the rule
View Orchestration Queue
Only Organization Administrators can see the orchestration queue. Note that all downloads (active or pending) within the organization will be displayed in the queue. That is: the queue is not limited to only show downloads from throttled networks.
To view the queue, log in as an Organization Administrator, then
- Click on the Organization tab
- Click on the Queue sub-tab
The queue shows active and pending downloads, along with the following information:
- Name - Solsta user initiating the download
- Device - Name of device (as specified in the user machine’s OS)
- Product - Name of Solsta product being installed/updated
- Environment - Name of Solsta environment being installed/updated
- Location - Shows the path on the machine where the environment is being installed to. This column is hidden by default, but can be displayed using the “Show Location” toggle box (see below)
- Progress - Progress of install or update (reported in 15 second intervals)
- Created - Timestamp for when the download reported to the Orchestration Service
From this screen, an Organization Administrator can click on the Show location box on the top left of the UI to toggle the Location field on/off for each download in the queue.
You can hover over the value of each row in the Location column to display the full path.
