flag

Flagging of the data. The selected flagging steps are executed in the same order in which they are given below.

enable

bool

Execute the flag worker.

field

{“target”, “calibrators”}, optional, default = calibrators

Fields that should be flagged. It can be set to either ‘target’ or ‘calibrators’ (i.e., all calibrators) as defined in the obsconf worker. Note that this selection is ignored – i.e., all fields in the selected .MS file(s) are flagged – in the flagging steps ‘flag_mask’ and ‘flag_Urange’ (see below). If a user wants to only flag a subset of the calibrators the selection can be further refined using ‘calfields’ below. The value of ‘field’ is also used to compose the name of the .MS file(s) that should be flagged, as explained in ‘label_in’ below.

label_in

str, optional, default = ‘ ‘

This label is added to the input .MS file(s) name, given in the getdata worker, to define the name of the .MS file(s) that should be flagged. These are <input>_<label>.ms if ‘field’ (see above) is set to ‘calibrators’, or <input>-<target>_<label>.ms if ‘field’ is set to ‘target’ (with one .MS file for each target in the input .MS). If empty, the original .MS is flagged with the field selection explained in ‘field’ above.

calfields

str, optional, default = auto

If ‘field’ above is set to ‘calibrators’, users can specify here what subset of calibrators to process. This should be a comma-separated list of ‘xcal’ ,’bpcal’, ‘gcal’ and/or ‘fcal’, which were all set by the obsconf worker. Alternatively, ‘auto’ selects all calibrators.

rewind_flags

Rewind flags to specified version.

enable

bool, optional, default = True

Enable the ‘rewind_flags’ segment.

mode

{“reset_worker”, “rewind_to_version”}, optional, default = reset_worker

If mode = ‘reset_worker’ rewind to the flag version before this worker if it exists, or continue if it does not exist; if mode = ‘rewind_to_version’ rewind to the flag version given by ‘version’ below.

version

str, optional, default = auto

Flag version to rewind to. If set to ‘auto’ it will rewind to the version prefix_workername_before, where ‘prefix’ is set in the ‘general’ worker, and ‘workername’ is the name of this worker including the suffix ‘__X’ if it is a repeated instance of this worker in the configuration file. Note that all flag versions that had been saved after this version will be deleted.

overwrite_flagvers

bool, optional, default = False

Allow CARACal to overwrite existing flag versions. Not recommended. Only enable this if you know what you are doing.

unflag

Unflag all visibilities for the selected field(s).

enable

bool, optional, default = False

Enable the ‘unflag’ segment.

flag_autopowerspec

Flags antennas based on drifts in the scan average of the auto-correlation spectra per field. This doesn’t strictly require any calibration. It is also not field-structure dependent, since it is just based on the DC of the field. Compares scan to median power of scans per field per channel. Also compares antenna to median of the array per scan per field per channel. This should catch any antenna with severe temperature problems.

enable

bool, optional, default = False

Enable the ‘flag_autopowerspec’ segment.

scan_thr

int, optional, default = 3

Threshold for flagging (in sigma) above the rest of the scans per field per channel.

ant_group_thr

int, optional, default = 5

Threshold for flagging (in sigma) above array median-power spectra per scan per field per channel.

col

str, optional, default = DATA

Data column to flag.

threads

int, optional, default = 8

Number of threads to use.

flag_autocorr

Flag auto-correlations, through the FLAGDATA task in CASA.

enable

bool, optional, default = False

Enable the ‘flag_autocorr’ segment.

flag_quack

Do quack flagging, i.e. flag the beginning and/or end chunks of each scan. Again, this is done through FLAGDATA.

enable

bool, optional, default = False

Enable the ‘flag_quack’ segment.

interval

float, optional, default = 8.

Time interval (in seconds) to flag.

mode

{“beg”, “endb”, “tail”, “end”}, optional, default = beg

Quack flagging mode. Options are ‘beg’ (which flags the beginning of the scan), ‘endb’ (which flags the end of the scan), ‘tail’ (which flags everything but the first specified seconds of the scan), and ‘end’ (which flags all but the last specified seconds of the scan).

flag_elevation

Use CASA FLAGDATA to flag antennas with pointing elevation outside the selected range.

enable

bool, optional, default = False

Enable the ‘flag_elevation’ segment.

low

float, optional, default = 0

Lower elevation limit. Antennas pointing at an elevation below this value are flagged.

high

float, optional, default = 90

Upper elevation limit. Antennas pointing at an elevation above this value are flagged.

flag_shadow

Use CASA FLAGDATA to flag shadowed antennas.

enable

bool, optional, default = False

Enable the ‘flag_shadow’ segment.

tol

float, optional, default = 0.

Amount of shadow allowed (in metres). A positive number allows antennas to overlap in projection. A negative number forces antennas apart in projection.

full_mk64

bool, optional, default = False

Consider all MeerKAT-64 antennas in the shadowing calculation, even if only a subarray is used.

flag_spw

Use CASA FLAGDATA to flag spectral windows/channels.

enable

bool, optional, default = False

Enable the ‘flag_spw’ segment.

chans

str, optional, default = *:856~880MHz , *:1658~1800MHz, *:1419.8~1421.3MHz

Channels to flag. Given as “spectral window index:start channel ~ end channel” e.g. “*:856~880MHz”. End channels are not inclusive.

ensure_valid

bool, optional, default = True

Check whether the channel selection returns any data. If it does not, FLAGDATA is not executed (preventing the pipeline from crashing). This check only works with the following spw formats (multiple, comma-separated selections allowed), “*:firstchan~lastchan”; “firstspw~lastspw:firstchan~lastchan”; “spw:firstchan~lastchan”; “firstchan~lastchan”. Channels are assumed to be in frequency (Hz, kHz, MHz, GHz allowed; if no units are given it assumes Hz).

flag_time

Use CASA FLAGDATA to flag a specified timerange in the data.

enable

bool, optional, default = False

Enable the ‘flag_time’ segment.

timerange

str, optional, default = ‘ ‘

Timerange to flag. Required in the format ‘YYYY/MM/DD/HH:MM:SS~YYYY/MM/DD/HH:MM:SS’.

ensure_valid

bool, optional, default = False

Check whether the timerange is in the MS being considered. This stops the pipeline from crashing when multiple datasets are being processed.

flag_scan

Use CASA FLAGDATA to flag bad scans.

enable

bool, optional, default = False

Enable the ‘flag_scan’ segment.

scans

str, optional, default = 0

Use CASA FLAGDATA syntax for selecting scans to flag.

flag_antennas

Flag bad antennas. Or just the ones you have sworn a vendetta against.

enable

bool, optional, default = False

Enable the ‘flag_antennas’ segment.

antennas

str, optional, default = 0

Use CASA FLAGDATA syntax for selecting antennas to flag.

timerange

str, optional, default = ‘ ‘

Timerange to flag. Required in the format ‘YYYY/MM/DD/HH:MM:SS~YYYY/MM/DD/HH:MM:SS’.

ensure_valid

bool, optional, default = False

Check whether the timerange is in the MS being considered. This stops the pipeline from crashing when multiple datasets are being processed.

flag_mask

Apply a static mask to flag known RFI in all fields of the selected MS file(s).

enable

bool, optional, default = False

Enable the ‘flag_mask’ segment.

mask

str

The mask to apply. This can be provided by the user, but CARACal also provides an existing static mask for MeerKAT, specify ‘meerkat.rfimask.npy’ to use it.

uvrange

str, optional, default = ‘ ‘

Select range in UV-distance (CASA-style range, e.g. ‘lower~upper’) for flagging. This is in units of metres. Leaving this parameter blank will select the entire range in UV-distance.

flag_manual

Manually flag subsets of data, using a syntax based on the CASA flagdata task

enable

bool, optional, default = False

Enable the ‘flag_manual’ segment.

rules

list of str, optional, default = ‘ ‘

Sequence of flagging rules, of the form “ms_pattern key:value key:value”

flag_rfi

Flag RFI using AOFlagger, Tricolour, or CASA FLAGDATA with tfcrop.

enable

bool, optional, default = False

Enable the ‘flag_rfi’ segment.

flagger

{“aoflagger”, “tricolour”, “tfcrop”}, optional, default = aoflagger

Choose flagger for automatic flagging. Options are ‘aoflagger’, ‘tricolour’ and ‘tfcrop’.

col

str, optional, default = DATA

Specify which column to flag.

aoflagger

strategy

str, optional, default = firstpass_Q.rfis

The AOFlagger strategy file to use.

ensure_valid

bool, optional, default = True

Ensure that the selected AOFlagger strategy is compatible with the type of correlations present in the input .MS file(s). E.g., attempts to flag on Stokes V for an .MS with XX and YY only will result in an error and CARACal exiting. The rules are 1. XY,YX must be present in order to flag on Stokes V,U (or on XY,YX), and 2. XX,YY must be present in order to flag on Stokes I,Q (or on XX,YY). Disable this parameter only if you know what you are doing.

readmode

{“indirect”, “memory”, “auto”}, optional, default = auto

AOflagger read mode. If set to ‘indirect’, AOflagger temporarily writes a reordered .MS file to disc, which results in fast flagging but requires free disc space. If set to ‘memory’, AOflagger reads the .MS file into memory, which is even faster than ‘indirect’ but is impossible for large files. If set to ‘auto’, AOflagger will decide between the ‘memory’ mode and the ‘direct’ mode – the slowest mode – in which AOFlagger reads baselines by scanning the entire file for the data relevant for the currently required baseline.

tricolour

backend

{“numpy”, “zarr-disk”}, optional, default = numpy

Visibility and flag data is re-ordered from an MS-row ordering into time-frequency windows ordered by baseline. Options are ‘numpy’ (if only a few scans worth of visibility data need to be re-ordered) and ‘zarr-disk’ (for larger data sizes, where re-ordering on disk, rather than in memory, is necessary).

mode

{“auto”, “manual”}, optional, default = auto

If mode is set to ‘manual’, Tricolour uses the flagging strategy set via ‘strategy’ below. If mode is set to ‘auto’, it uses the strategy in ‘strategy_narrowband’ in case of small bandwidth of the .MS file(s).

strategy

str, optional, default = mk_rfi_flagging_calibrator_fields_firstpass.yaml

Name of the Tricolour strategy file.

strat_narrow

str, optional, default = calibrator_mild_flagging.yaml

Name of the Tricolour strategy file to be used for an MS with narrow bandwidth, if mode = ‘auto’ (see above).

tfcrop

usewindowstats

{“none”, “sum”, “std”, “both”}, optional, default = std

Calculate additional flags using sliding-window statistics. Options are ‘none’, ‘sum’, ‘std’, and ‘both’. See usewindowstats, within documentation for CASA FLAGDATA, for further details.

combinescans

bool, optional, default = False

Accumulate data across scans, depending on the value set for the ‘ntime’ parameter.

flagdimension

{“freq”, “time”, “freqtime”, “timefreq”}, optional, default = freqtime

Dimension(s) along which to calculate a fit(/fits) to the data. Options are ‘freq’, ‘time’, ‘freqtime’, and ‘timefreq’. Note that the differences between ‘freqtime’ and ‘timefreq’ are only apparent if RFI in one dimension is signifcantly stronger than in the other.

timecutoff

float, optional, default = 4.0

Flagging threshold, in units of standard deviation (i.e. sigma) from the fit along the time axis.

freqcutoff

float, optional, default = 3.0

Flagging threshold, in units of standard deviation (i.e. sigma) from the fit along the frequency axis.

correlation

str, optional, default = ‘ ‘

Specify the correlation(s) to be considered for flagging with ‘tfcrop’. E.g. ‘XX,YY’, or leave as ‘’ to select all correlations.

inspect

Use the diagnostic products of RFInder to inspect the presence of RFI.

enable

bool, optional, default = False

Enable the ‘inspect’ segment.

telescope

{“meerkat”, “apertif”, “wsrt”}, optional, default = meerkat

Name of the telescope. Options are ‘meerkat’, ‘apertif’, and ‘wsrt’.

field

str, optional, default = target

Field over which to determine flag statistics. Options are ‘gcal’, ‘bpcal’, and ‘target’.

polarization

{“xx”, “XX”, “yy”, “YY”, “xy”, “XY”, “yx”, “YX”, “q”, “Q”}, optional, default = q

Select the polarization, e.g. ‘xx’, ‘yy’, ‘xy’, ‘yx’, ‘q’ (and also each of these in upper case).

spw_enable

bool, optional, default = True

Enable averaging/rebinning in frequency.

spw_width

int, optional, default = 10

Frequency width of rebinned output table (in units of MHz).

time_enable

bool, optional, default = True

Enable averaging/rebinning in time.

time_step

int, optional, default = 5

Time steps (in units of minutes).

summary

Use CASA FLAGDATA in ‘summary’ mode to write flagging summary at the end of the pre-calibration flagging.

enable

bool, optional, default = True

Enable the ‘summary’ segment.

report

bool, optional, default = False

(Re)generate a full HTML report at the end of this worker.

cabs

list of map, optional, default = ‘ ‘

Specifies non-default image versions and/or tags for Stimela cabs. Running with scissors: use with extreme caution.