EIR-OPS-034: NACK/Timeout/Unexpected TM


Objective

To determine why EIRSAT-1 is not responding to TCs as expected.


Introduction

Using this procedure, the Operator will try to determine why EIRSAT-1 is not responding to TCs as expected. This procedure takes into account three possible scenarios where the expected TM is not received from the spacecraft.

By determining the cause for the NACK/Timeout/unexpected TM, this procedure should also help Operators to overcome the issue.


Procedure

The following three scenarios are considered:

  • A. Unexpected TM - TM is returned from the spacecraft in response to TCs with a positive acknowledgment (i.e. an ‘ACK’). However, the TM returned is not as expected. For example, let’s say we justed invoked the mission.ModeManager.transitionToCommissioning action but the mission.ModeManager.Mode parameter is not returning ‘01’ (Commissioning Mode).

  • B. NACK - TM is returned in response to TCs but rather than a TC acknowledgment (i.e. an ‘ACK’), the spacecraft is returning a negative acknowledge (i.e. a ‘NACK’) indicating that the operation associated with the TC failed due to some error.

  • C. Timeout - No TM at all is returned from the spacecraft in response to TCs.

The Operator should skip to the section that is relevant to their circumstances.

Important

As this is a failure/unexpected scenario, this procedure is a guide of the questions to consider rather than a strict set of instructions.


A. Unexpected TM

A.1.

  • If the spacecraft returns TM to a TC that is not consistent with an expected/allowed response to that TC, the first thing the Operator should do is verify that the correct spacecraft database (SCDB) is loaded into MCS. As the spacecraft may have rebooted into a different image since the last communication pass (or even since the last TC), the Operator should Get the platform.obc.OBC.currBootImage parameter to confirm that the SCDB matches the currently operating image.

Note

The platform.obc.OBC.currBootImage parameter should have the same parameter ID in all SCDBs so the spacecraft should respond no matter what SCDB is loaded into MCS when sending the TC.

TC Details

MCS Operation

Get

Action/Param Name

platform.obc.OBC.currBootImage

Data Expected with TC

No

TM Details

Data Expected from TC

currBootImage ( + ACK )

Data Size

Integer 8 (1 hex value)

Data Info

Index of currently executing softwware image

Allowed Value(s)

0, 1 or 2

A.2.

  • For further verification is needed, the Operator should also consult with the Software Engineer to ensure that the correct versions of the SCDB are being loaded into MCS by the Operators.



B. NACK

B.1.

  • If the spacecraft returns a NACK, we at least know that the TC sent by the GS is getting accepted by the spacecraft’s CMC and is then being passed to the OBC as a valid space packet received over RF. The issue is instead related to the execution of the TC by the OBC. Therefore, the Operator should consider the following:

    • Is the NACK a once-off (i.e. does the same TC get an ‘ACK’ response when sent a second time)? While receiving NACKs from the spacecraft is not ideal, infrequent NACK responses to TCs is acceptable/expected.

    • If NACKs are consistently returned by the spacecraft, are NACKs received for only this/certain TCs or all TCs (e.g. is the response to Get the Version.satelliteString a NACK)?

    • If NACKs are only returned for certain commands, the Operator should determine if the correct SCDB is loaded into MCS? To gain confidence in the SCDB being used the Operator should do either of the below:

      • Use the data from the beacon automatically transmitted by the spacecraft to determine the current boot image (i.e. the platform.obc.OBC.currBootImage parameter) and/or

      • Get the platform.obc.OBC.currBootImage parameter to confirm that the SCDB matches the current operating image.

    Note

    The platform.obc.OBC.currBootImage parameter should have the same parameter ID in all SCDBs so the spacecraft should respond no matter what SCDB is loaded into MCS when sending the TC.

    TC Details

    MCS Operation

    Get

    Action/Param Name

    platform.obc.OBC.currBootImage

    Data Expected with TC

    No

    TM Details

    Data Expected from TC

    currBootImage ( + ACK )

    Data Size

    Integer 8 (1 hex value)

    Data Info

    Index of currently executing softwware image

    Allowed Value(s)

    0, 1 or 2

  • If the correct SCDB is loaded into MCS, the Operator should confirm that the data being sent with the TC ‘valid’? e.g.:

    • If a parameter is a multi-row parameter, when the parameter window is opened on MCS, the Operator should either be able to Query the number of rows in the parameter (if the parameter has a variable number of rows) or the fixed number of rows in the parameter should be displayed next to the greyed-out Query button. If the Operator tries to Get or Set more than than the number of parameter rows available, a NACK will be sent by the OBC as the Get / Set data was invalid.

    • If a parameter or action argument has allowed value ranges (which are hard-coded into the OBSW or subsystem firmwares), a NACK may be returned if these ranges are exceeded.

    • If Invoke is called with an action argument larger/smaller than the max/min number of bytes allowed for the action argument, a NACK will be returned.

  • An event is commonly raised with a NACK. If a communication pass was started (i.e. via the EIR-OPS-003: Start a Communication Pass procedure) events should be transmitted live by the spacecraft with the NACK. Alternatively, the Event log can be downlinked from the spacecraft (i.e. via the EIR-OPS-011: Downlink Data From Storage procedure). The Operator should assess if/what event was raised to determine if it offers any useful information on the reason for the NACK response.



C. Timeout

C.1.

  • If instead no response is at all observed to TCs from the UCD GS. The Operator should confirm the following:

    • Some TCs take time to execute. Although the parameter/action window on MCS may indicate that a timeout has occurred, the Operator should still verify using MCS’s Packet Monitor that no response (i.e. no ACK/NACK) is eventually returned.

    • Has the ‘hardware’ beacon (i.e. the beacon automatically transmitted by the CMC when OBC-to-CMC communications fail) been observed by either the UCD GS and/or the amateur radio community? If this beacon is the ONLY TM being received from the spacecraft, the Operator should:

      • Consider using the EIR-OPS-031: DTMF Reboot procedure to reset the OBC, or/and

      • Discuss if/when the OBSW’s CMCAliveCheck functionality is expected to reset the OBC.

    • Is any of the data automatically transmitted by the spacecraft being observed/processed/parsed by the UCD GS? If it is, we know that at the very least the Rx chain of the GS is operational and so, the following steps should be taken:

      • Confirm that MCS’s connection settings are correct, and that the bottom right-hand side of the main MCS GUI indicates the connection is enabled.

      • Confirm that MCS is adding authentication framing to the TCs as this is expected by the spacecraft:

        Tip

        Although authentication framing must be added to TCs, a correct HMAC key and sequence number are only relevant if TC authentication is enabled. Therefore, dismiss the following two bullet points if TC authentication is known to be disabled at this time.

        • Confirm with the Software and/or Systems Engineer that the HMAC key being used is correct (for security reasons, this HMAC key will not be saved on-line/distributed to the full team).

        • Confirm that the Sequence number (i.e. the ACK’d TC counter) matches the sequence number maintained by the OBSW. Beacon data, automatically transmitted by the spacecraft, should be used to determine the up-to-date OBSW sequence number (i.e. the comms.HMAC.sequenceNumber parameter).

      • Assess whether a fault has occurred in the Tx chain of the GS (e.g. compare the format of a TC sent out from the GS to the expected format for that command).

      • Confirm that the GS transmission power is as expected.

    • If data being automatically transmitted by the spacecraft is instead observed by ONLY the amateur radio community (and not by the UCD GS when a pass is expected to be on-going), assess whether a larger fault has occurred in the full GS system.

    • If data being automatically transmitted by the spacecraft in not observed by ANY ground stations (i.e. complete silence from spacecraft), consider the possibility that RF transmission from the spacecraft has become disabled. In this case, the Operator should consider sending Set TCs to the spacecraft’s predicted location on the sky to enabled the comms.TMTCBuffer.spacelinkTxEnable parameter.

    Important

    As this parameter has a different parameter ID in different SCDBs, the Operator should send this TC using the SCDB for both the failsafe and primary images.

    Danger

    If antenna deployment/initial AOS has NOT YET been confirmed/achieved, take extreme care prior to enabling RF transmission - if the UHF antenna elements are not fully deployed, RF transmissions could damage the CMC. Have you allowed enough time for the possibility of (multiple) reboots during the post-launch wait to ADM burns and/or for repeated antenna deployment attempts?

    TC Details

    MCS Operation

    Set

    Action/Param Name

    comms.TMTCBuffer.spacelinkTxEnable

    Data Expected with TC

    Yes

    Data Size

    1 byte

    Data Info

    Whether RF transmission is enabled (1) or disabled (0)

    Allowed Value(s)

    0 - 1

    Expected Value(s)

    1

    TM Details

    Data Expected from TC

    No ( + ACK )

END OF PROCEDURE