EIR-OPS-031: DTMF Reboot


Objective

To power-cycle the spacecraft via the DTMF reset tone in an effort to re-establish 2-way communications.


Introduction

Danger

This procedure should be used when no TM is received from the spacecraft by either the UCD GS or the amateur radio community and in this case, only after the Operator has already tried to re-enable RF Tx on the spacecraft as detailed in the NACK/Timeout/Unexpected TM procedure. Alternatively, this procedure should be used if the only TM being received from the spacecraft is TM from the CMC’s hardware beacon.

Using this procedure, the Operator will work to re-establish 2-way communications with the spacecraft following the DTMF reset tone.


Procedure

This procedure contains the following sub-procedures:

A. Sending the DTMF Reset Tone
B. Re-establishing 2-way Communications
C. Downlinking Data
D. Debugging the Issue
E. Resuming Nominal Operations

Note

A communication window with the spacecraft is required for Sections A, B, C and E.


A. Sending the DTMF Reset Tone

A.1.

  • The Operator should consult with the GS Team about transmitting the DTMF reset tones to the spacecraft.

  • When the tones have been sent, proceed to Section B.

Note

Sending the DTMF tones is not always successfully at resetting the OBC. Therefore, multiple attempts can be made to send the tones prior to proceeding to Section B. Alternatively, attempt Section B and return to this section if 2-way communication is not promptly established or if the previous DTMF reset attempt is believed to have failed.



B. Re-establishing 2-way Communications

Note

As the current state of the spacecraft is potentially uncertain after the fault that led to loss of communications and then the DTMF reset, it is advised that the Operator carry out the below steps to re-establish the state of the spacecraft prior to proceeding the Section C.

B.1.

  • Get the Version.satelliteString parameter.

  • Repeat sending this Get TC (as long as the pass proceeds) until TM is received from EIRSAT-1.

TC Details

MCS Operation

Get

Action/Param Name

Version.satelliteString

Data Expected with TC

No

TM Details

Data Expected from TC

satelliteString ( + ACK )

Data Size

8 bytes

Data Info

the “EIRSAT-1” string.

Allowed Value(s)

E I R S A T - 1

Expected Value(s)

E I R S A T - 1

B.2.

  • To determine which mission image is the current boot image on the OBC, Get the platform.OBC.obc.currBootImage parameter.

TC Details

MCS Operation

Get

Action/Param Name

platform.OBC.obc.currBootImage

Data Expected with TC

No

TM Details

Data Expected from TC

currBootImage ( + ACK )

Data Size

1 byte

Data Info

the current boot image of EIRSAT-1

Allowed Value(s)

00 - 02 (Hex)

Where…

currBootImage

Image

0

failsafe

1

primary1

2

primary2

B.3.

  • To determine which state the Separation Sequence is currently in, Get the mission.SeparationSequence.state parameter.

TC Details

MCS Operation

Get

Action/Param Name

mission.SeparationSequence.state

Data Expected with TC

No

TM Details

Data Expected from TC

state ( + ACK )

Data Size

1 byte

Data Info

Current state of the Separation Sequence

Allowed Value(s)

00 - 09 or 66 (dec)

Expected Value(s)

66 (i.e. the finished state), if initial AOS is complete

Where…

state

State

00

Init

01

PostLaunchWait

02

MinYPrimaryBurnStart

03

PlusYPrimaryBurnStart

04

MinXPrimaryBurnStart

05

PlusXPrimaryBurnStart

06

BurnTimeWait

07

BurnOff

08

BetweenBurnWait

09

SecondaryBurnStart

42

Finish

B.4.

  • If currBootImage = 0 in Step B.2, skip ahead to Section C as the failsafe image does not have a Mode Manager component.

  • Else, Get the mission.ModeManager.Mode parameter to determine which mode EIRSAT-1 is currently operating in.

TC Details

MCS Operation

Get

Action/Param Name

mission.ModeManager.Mode

Data Expected with TC

No

TM Details

Data Expected from TC

Mode ( + ACK )

Data Size

1 byte

Data Info

Current Operational Mode of the satellite

Allowed Value(s)

0 - 4

Where …

Mode

Operational Mode

0

Separation Sequence

1

Commissioning

2

Nominal

3

WBC

4

Safe


C. Downlinking Data

C.1.

  • NEWEST rows of data from the Event , HK and TED data logs should initially be given the highest priority for data downlink.

  • However, the priority for data downlink should be re-assessed by the Operators between each pass following the analyses performed in Section D, as the analyses will likely reveal that a particular data type may be more relevant than another to fully establish the cause of this anomaly.

  • With the data downlink priority for a given pass established, during the pass, downlink data not previously retrieved by the ground segment from the relevant on-board storage channels according to the EIR-OPS-011: Downlink Data From Storage procedure.



D. Debugging the Issue

Note

As this is a failure/unexpected scenario, rather than a strict set of instructions, this section of the procedure instead aims to help guide the Operator in their fault analyses. The steps/analyses carried out in practice should be tailored based on the Operators findings.

Warning

TC authentication is disabled at boot to reduce the risk of loosing communication with the spacecraft. Therefore, TC authentication is now disabled after the DTMF reboot. The Operator should consider following the EIR-OPS-009: Enable TC Authentication procedure to re-enable TC authentication to prevent replay attacks.

D.1.

  • Assess the most recent data generated/logged by the OBC and downlinked in Section C to:

    • Determine if any TCs were accepted by the spacecraft that were not sent by the UCD GS.

    • Determine if the OBC was generating and logging data right up until the DTMF reset or if data generation/logging stopped at an earlier point in time.

    • Determine if reboots prior to the DTMF reset occurred whose timings are consistent with the time at which 2-way communications was lost.

    • Determine if any anomalous behavior is seen in the data prior to, or at the time at which, 2-way communications was lost. In addition to instantaneous values of each parameter in the downlinked data, check how the values changed with time. Use the Grafana Mission Monitoring System (MMS) to help with this. Pay particular attention to CMC-related parameters.

    • …

Note

If the hardware beacon is detected when communications are re-established, it is expected that there will be several logged events since last communication due to the OBC and CMC not being able to communicate with each other (i.e. the issue that led to the hardware beacon) and generating events.


D.2.

  • The Operators should consider if a time-related bug is potentially the cause. More specifically, the Operators should consider:

    • Has the spacecraft remained powered ON without interruption for this long of a duration before?

    • Has an image remained the boot image without interruption for this long of a duration before?

    • …


…



E. Resuming Nominal Operations

E.1.


END OF PROCEDURE