meerkat package

Meerkat.

class AbstractCell(*args, **kwargs)

Bases: ABC

get(*args, **kwargs) → object

Get me the thing that this cell exists for.

loader(*args, **kwargs) → object

property metadata: dict

Get the metadata associated with this cell.

class AbstractColumn(data: Sequence | None = None, collate_fn: Callable | None = None, formatter: Callable | None = None, *args, **kwargs)

Bases: BlockableMixin, CloneableMixin, CollateMixin, ColumnIOMixin, FunctionInspectorMixin, LambdaMixin, MappableMixin, MaterializationMixin, ProvenanceMixin, ABC

An abstract class for Meerkat columns.

append(column: AbstractColumn) → None

batch(batch_size: int = 1, drop_last_batch: bool = False, collate: bool = True, num_workers: int = 0, materialize: bool = True, *args, **kwargs)

Batch the column.

Parameters:

• batch_size – integer batch size

• drop_last_batch – drop the last batch if its smaller than batch_size

• collate – whether to collate the returned batches

Returns:

batches of data

static concat(columns: Sequence[AbstractColumn]) → None

filter(function: Callable, with_indices=False, input_columns: str | List[str] | None = None, is_batched_fn: bool = False, batch_size: int | None = 1, drop_last_batch: bool = False, num_workers: int | None = 0, materialize: bool = True, pbar: bool = False, **kwargs) → AbstractColumn | None

Filter the elements of the column using a function.

classmethod from_data(data: Columnable | AbstractColumn)

Convert data to a meerkat column using the appropriate Column type.

full_length()

classmethod get_writer(mmap: bool = False, template: AbstractColumn | None = None)

head(n: int = 5) → AbstractColumn

Get the first n examples of the column.

is_equal(other: AbstractColumn) → bool

Tests whether two columns.

Parameters:

other (AbstractColumn) – [description]

streamlit()

tail(n: int = 5) → AbstractColumn

Get the last n examples of the column.

to_pandas() → Series

Columnable

alias of Union[Sequence, ndarray, Series, Tensor]

property data

Get the underlying data.

property formatter: Callable

property is_mmap

logdir: Path = PosixPath('/home/docs/meerkat')

property metadata

class ArrowArrayColumn(data: Sequence, *args, **kwargs)

Bases: AbstractColumn

block_class

alias of ArrowBlock

classmethod concat(columns: Sequence[ArrowArrayColumn])

is_equal(other: AbstractColumn) → bool

Tests whether two columns.

Parameters:

other (AbstractColumn) – [description]

to_numpy()

to_pandas()

to_tensor()

class AudioColumn(data: Sequence[str] | None = None, transform: callable | None = None, loader: callable | None = None, base_dir: str | None = None, *args, **kwargs)

Bases: FileColumn

A lambda column where each cell represents an audio file on disk. The underlying data is a PandasSeriesColumn of strings, where each string is the path to an image. The column materializes the images into memory when indexed. If the column is lazy indexed with the lz indexer, the images are not materialized and an FileCell or an AudioColumn is returned instead.

Parameters:

• data (Sequence[str]) – A list of filepaths to images.

• transform (callable) –

  A function that transforms the image (e.g. torchvision.transforms.functional.center_crop).

  Warning

  In order for the column to be serializable, the transform function must be pickleable.

• loader (callable) –

  A callable with signature def loader(filepath: str) -> PIL.Image:. Defaults to torchvision.datasets.folder.default_loader.

  Warning

  In order for the column to be serializable with write(), the loader function must be pickleable.

• base_dir (str) – A base directory that the paths in data are relative to. If None, the paths are assumed to be absolute.

collate(batch)

Collate data.

classmethod default_loader(*args, **kwargs)

class CellColumn(cells: Sequence[AbstractCell] | None = None, *args, **kwargs)

Bases: AbstractColumn

static concat(columns: Sequence[CellColumn])

classmethod from_cells(cells: Sequence[AbstractCell], *args, **kwargs)

is_equal(other: AbstractColumn) → bool

Tests whether two columns.

Parameters:

other (AbstractColumn) – [description]

property cells

class DataPanel(data: dict | list | Dataset | None = None, *args, **kwargs)

Bases: CloneableMixin, FunctionInspectorMixin, LambdaMixin, MappableMixin, MaterializationMixin, ProvenanceMixin

Meerkat DataPanel class.

add_column(name: str, data: Sequence | ndarray | Series | Tensor, overwrite=False) → None

Add a column to the DataPanel.

append(dp: DataPanel, axis: str | int = 'rows', suffixes: Tuple[str] = None, overwrite: bool = False) → DataPanel

Append a batch of data to the dataset.

example_or_batch must have the same columns as the dataset (regardless of what columns are visible).

batch(batch_size: int = 1, drop_last_batch: bool = False, num_workers: int = 0, materialize: bool = True, shuffle: bool = False, *args, **kwargs)

Batch the dataset. TODO:

Parameters:

• batch_size – integer batch size

• drop_last_batch – drop the last batch if its smaller than batch_size

Returns:

batches of data

consolidate()

filter(function: Callable | None = None, with_indices=False, input_columns: str | List[str] | None = None, is_batched_fn: bool = False, batch_size: int | None = 1, drop_last_batch: bool = False, num_workers: int = 0, materialize: bool = True, pbar: bool = False, **kwargs) → DataPanel | None

Filter operation on the DataPanel.

classmethod from_arrow(table: Table)

Create a Dataset from a pandas DataFrame.

classmethod from_batch(batch: Dict[str, List | AbstractColumn]) → DataPanel

Convert a batch to a Dataset.

classmethod from_batches(batches: Sequence[Dict[str, List | AbstractColumn]]) → DataPanel

Convert a list of batches to a dataset.

classmethod from_csv(filepath: str, *args, **kwargs)

Create a Dataset from a csv file.

Parameters:

• filepath (str) – The file path or buffer to load from. Same as pandas.read_csv().

• *args – Argument list for pandas.read_csv().

• **kwargs – Keyword arguments for pandas.read_csv().

Returns:

The constructed datapanel.

Return type:

DataPanel

classmethod from_dict(d: Dict) → DataPanel

Convert a dictionary to a dataset.

Alias for Dataset.from_batch(..).

classmethod from_feather(path: str)

Create a Dataset from a feather file.

classmethod from_huggingface(*args, **kwargs)

Load a Huggingface dataset as a DataPanel.

Use this to replace datasets.load_dataset, so

>>> dict_of_datasets = datasets.load_dataset('boolq')

becomes

>>> dict_of_datapanels = DataPanel.from_huggingface('boolq')

classmethod from_jsonl(json_path: str) → DataPanel

Load a dataset from a .jsonl file on disk, where each line of the json file consists of a single example.

classmethod from_pandas(df: DataFrame)

Create a Dataset from a pandas DataFrame.

head(n: int = 5) → DataPanel

Get the first n examples of the DataPanel.

items()

keys()

map(function: Callable | None = None, with_indices: bool = False, input_columns: str | List[str] | None = None, is_batched_fn: bool = False, batch_size: int | None = 1, drop_last_batch: bool = False, num_workers: int = 0, output_type: type | Dict[str, type] = None, mmap: bool = False, mmap_path: str = None, materialize: bool = True, pbar: bool = False, **kwargs) → Dict | List | AbstractColumn | None

merge(right: DataPanel, how: str = 'inner', on: str | List[str] | None = None, left_on: str | List[str] | None = None, right_on: str | List[str] | None = None, sort: bool = False, suffixes: Sequence[str] = ('_x', '_y'), validate=None)

classmethod read(path: str, *args, **kwargs) → DataPanel

Load a DataPanel stored on disk.

remove_column(column: str) → None

Remove a column from the dataset.

streamlit()

tail(n: int = 5) → DataPanel

Get the last n examples of the DataPanel.

to_jsonl(path: str) → None

Save a Dataset to a jsonl file.

to_pandas() → DataFrame

Convert a Dataset to a pandas DataFrame.

update(function: Callable | None = None, with_indices: bool = False, input_columns: str | List[str] | None = None, is_batched_fn: bool = False, batch_size: int | None = 1, remove_columns: List[str] | None = None, num_workers: int = 0, output_type: type | Dict[str, type] = None, mmap: bool = False, mmap_path: str = None, materialize: bool = True, pbar: bool = False, **kwargs) → DataPanel

Update the columns of the dataset.

values()

write(path: str) → None

Save a DataPanel to disk.

property columns

Column names in the DataPanel.

property data: BlockManager

Get the underlying data (excluding invisible rows).

To access underlying data with invisible rows, use _data.

logdir: Path = PosixPath('/home/docs/meerkat')

property ncols

Number of rows in the DataPanel.

property nrows

Number of rows in the DataPanel.

property shape

Shape of the DataPanel (num_rows, num_columns).

class FileCell(transform: callable | None = None, loader: callable | None = None, data: str | None = None, base_dir: str | None = None)

Bases: FileLoaderMixin, LambdaCell

property absolute_path

class FileColumn(data: Sequence[str] | None = None, transform: callable | None = None, loader: callable | None = None, base_dir: str | None = None, *args, **kwargs)

Bases: FileLoaderMixin, LambdaColumn

A column where each cell represents an file stored on disk or the web. The underlying data is a PandasSeriesColumn of strings, where each string is the path to a file. The column materializes the files into memory when indexed. If the column is lazy indexed with the lz indexer, the files are not materialized and a FileCell or a FileColumn is returned instead.

Parameters:

• data (Sequence[str]) – A list of filepaths to images.

• transform (callable) –

  A function that transforms the image (e.g. torchvision.transforms.functional.center_crop).

  Warning

  In order for the column to be serializable, the transform function must be pickleable.

• loader (callable) –

  A callable with signature def loader(filepath: str) -> PIL.Image:. Defaults to torchvision.datasets.folder.default_loader.

  Warning

  In order for the column to be serializable with write(), the loader function must be pickleable.

• base_dir (str) – A base directory that the paths in data are relative to. If None, the paths are assumed to be absolute.

classmethod default_loader(*args, **kwargs)

classmethod from_filepaths(filepaths: Sequence[str], loader: callable | None = None, transform: callable | None = None, base_dir: str | None = None, *args, **kwargs)

is_equal(other: AbstractColumn) → bool

Tests whether two columns.

Parameters:

other (AbstractColumn) – [description]

class ImageColumn(data: Sequence[str] | None = None, transform: callable | None = None, loader: callable | None = None, base_dir: str | None = None, *args, **kwargs)

Bases: FileColumn

A column where each cell represents an image stored on disk. The underlying data is a PandasSeriesColumn of strings, where each string is the path to an image. The column materializes the images into memory when indexed. If the column is lazy indexed with the lz indexer, the images are not materialized and an ImageCell or an ImageColumn is returned instead.

Parameters:

• data (Sequence[str]) – A list of filepaths to images.

• transform (callable) –

  A function that transforms the image (e.g. torchvision.transforms.functional.center_crop).

  Warning

  In order for the column to be serializable, the transform function must be pickleable.

• loader (callable) –

  A callable with signature def loader(filepath: str) -> PIL.Image:. Defaults to torchvision.datasets.folder.default_loader.

  Warning

  In order for the column to be serializable with write(), the loader function must be pickleable.

• base_dir (str) – A base directory that the paths in data are relative to. If None, the paths are assumed to be absolute.

classmethod default_loader(*args, **kwargs)

class LambdaCell(fn: callable | None = None, data: any | None = None)

Bases: AbstractCell

get(*args, **kwargs)

Get me the thing that this cell exists for.

property data: object

Get the data associated with this cell.

class LambdaColumn(data: DataPanel | AbstractColumn, fn: callable | None = None, output_type: type | None = None, *args, **kwargs)

Bases: AbstractColumn

static concat(columns: Sequence[LambdaColumn])

fn(data: object)

Subclasses like ImageColumn should be able to implement their own version.

is_equal(other: AbstractColumn) → bool

Tests whether two columns.

Parameters:

other (AbstractColumn) – [description]

class ListColumn(data: Sequence | None = None, *args, **kwargs)

Bases: AbstractColumn

batch(batch_size: int = 1, drop_last_batch: bool = False, collate: bool = True, *args, **kwargs)

Batch the column.

Parameters:

• batch_size – integer batch size

• drop_last_batch – drop the last batch if its smaller than batch_size

• collate – whether to collate the returned batches

Returns:

batches of data

classmethod concat(columns: Sequence[ListColumn])

default_formatter()

classmethod from_list(data: Sequence)

is_equal(other: AbstractColumn) → bool

Tests whether two columns.

Parameters:

other (AbstractColumn) – [description]

class MedicalVolumeCell(paths: str | Path | PathLike | Sequence[str | Path | PathLike], loader: Callable | None = None, transform: Callable | None = None, cache_metadata: bool = False, *args, **kwargs)

Bases: PathsMixin, AbstractCell

Interface for loading medical volume data.

Examples

# Specify xray dicoms with default orientation ("SI", "AP"): >>> cell = MedicalVolumeCell(“/path/to/xray.dcm”, loader=DicomReader(group_by=None, default_ornt=(“SI”, “AP”))

# Load multi-echo MRI volumes >>> cell = MedicalVolumeCell(“/path/to/mri/scan/dir”, loader=DicomReader(group_by=”EchoNumbers”))

clear_metadata()

classmethod default_loader(paths: Sequence[Path], *args, **kwargs)

classmethod from_state(state, *args, **kwargs)

get(*args, cache_metadata: bool | None = None, **kwargs)

Get me the thing that this cell exists for.

get_metadata(ignore_bytes: bool = False, readable: bool = False, as_raw_type: bool = False, force_load: bool = False) → Dict

get_state()

class MedicalVolumeColumn(*args, **kwargs)

Bases: CellColumn

classmethod from_filepaths(filepaths: Sequence[str] | None = None, loader: callable | None = None, transform: callable | None = None, *args, **kwargs)

class NumpyArrayColumn(data: Sequence, *args, **kwargs)

Bases: AbstractColumn, NDArrayOperatorsMixin

block_class

alias of NumpyBlock

classmethod concat(columns: Sequence[NumpyArrayColumn])

classmethod from_array(data: ndarray, *args, **kwargs)

classmethod from_npy(path, mmap_mode=None, allow_pickle=False, fix_imports=True, encoding='ASCII')

classmethod get_writer(mmap: bool = False, template: AbstractColumn | None = None)

is_equal(other: AbstractColumn) → bool

Tests whether two columns.

Parameters:

other (AbstractColumn) – [description]

to_pandas() → Series

to_tensor() → Tensor

Use column.to_tensor() instead of torch.tensor(column), which is very slow.

property is_mmap

class PandasSeriesColumn(data: Sequence | None = None, collate_fn: Callable | None = None, formatter: Callable | None = None, *args, **kwargs)

Bases: AbstractColumn, NDArrayOperatorsMixin

block_class

alias of PandasBlock

cat

alias of _MeerkatCategoricalAccessor

dt

alias of _MeerkatCombinedDatetimelikeProperties

str

alias of _MeerkatStringMethods

classmethod concat(columns: Sequence[PandasSeriesColumn])

classmethod from_array(data: ndarray, *args, **kwargs)

is_equal(other: AbstractColumn) → bool

Tests whether two columns.

Parameters:

other (AbstractColumn) – [description]

to_pandas() → Series

to_tensor() → Tensor

Use column.to_tensor() instead of torch.tensor(column), which is very slow.

class SpacyColumn(data: Sequence[spacy_tokens.Doc] = None, *args, **kwargs)

Bases: ListColumn

classmethod from_docs(data: Sequence[spacy_tokens.Doc], *args, **kwargs)

classmethod from_texts(texts: Sequence[str], lang: str = 'en_core_web_sm', *args, **kwargs)

classmethod read(path: str, nlp: spacy.language.Language = None, lang: str = None, *args, **kwargs) → SpacyColumn

write(path: str, **kwargs) → None

property docs

property tokens

class TensorColumn(data: Sequence | None = None, *args, **kwargs)

Bases: NDArrayOperatorsMixin, AbstractColumn

block_class

alias of TensorBlock

classmethod concat(columns: Sequence[TensorColumn])

classmethod from_data(data: Sequence | ndarray | Series | Tensor | AbstractColumn)

Convert data to an EmbeddingColumn.

classmethod get_writer(mmap: bool = False, template: AbstractColumn | None = None)

is_equal(other: AbstractColumn) → bool

Tests whether two columns.

Parameters:

other (AbstractColumn) – [description]

to_pandas() → Series

to_tensor() → Tensor

class VideoColumn(*args, **kwargs)

Bases: CellColumn

Interface for creating a CellColumn from VideoCell objects.

classmethod from_filepaths(filepaths: Sequence[str] | None = None, time_dim: int | None = 1, transform: Callable | None = None, *args, **kwargs)

class provenance(enabled: bool = True)

Bases: object

concat(objs: Sequence[DataPanel] | Sequence[AbstractColumn], axis: str | int = 'rows', suffixes: Tuple[str] = None, overwrite: bool = False) → DataPanel | AbstractColumn

Concatenate a sequence of columns or a sequence of DataPanel`s. If sequence is empty, returns an empty `DataPanel.

• If concatenating columns, all columns must be of the same type (e.g. all

ListColumn). - If concatenating `DataPanel`s along axis 0 (rows), all `DataPanel`s must have the same set of columns. - If concatenating `DataPanel`s along axis 1 (columns), all `DataPanel`s must have the same length and cannot have any of the same column names.

Parameters:

• objs (Union[Sequence[DataPanel], Sequence[AbstractColumn]]) – sequence of columns or DataPanels.

• axis (Union[str, int]) – The axis along which to concatenate. Ignored if concatenating columns.

Returns:

concatenated DataPanel or column

Return type:

Union[DataPanel, AbstractColumn]

merge(left: DataPanel, right: DataPanel, how: str = 'inner', on: str | List[str] = None, left_on: str | List[str] = None, right_on: str | List[str] = None, sort: bool = False, suffixes: Sequence[str] = ('_x', '_y'), validate=None)

Subpackages

• meerkat.block package
  • Submodules
  • meerkat.block.abstract module
    • AbstractBlock
      • AbstractBlock.consolidate()
      • AbstractBlock.from_block_data()
      • AbstractBlock.from_column_data()
      • AbstractBlock.read()
      • AbstractBlock.write()
      • AbstractBlock.is_mmap
      • AbstractBlock.signature
    • BlockView
      • BlockView.block
      • BlockView.block_index
      • BlockView.data
  • meerkat.block.arrow_block module
    • ArrowBlock
      • ArrowBlock.Signature
      • ArrowBlock.from_block_data()
      • ArrowBlock.from_column_data()
      • ArrowBlock.signature
  • meerkat.block.manager module
    • BlockManager
      • BlockManager.add_column()
      • BlockManager.apply()
      • BlockManager.consolidate()
      • BlockManager.copy()
      • BlockManager.from_dict()
      • BlockManager.get_block_ref()
      • BlockManager.read()
      • BlockManager.remove()
      • BlockManager.reorder()
      • BlockManager.update()
      • BlockManager.view()
      • BlockManager.write()
      • BlockManager.ncols
      • BlockManager.nrows
  • meerkat.block.numpy_block module
    • NumpyBlock
      • NumpyBlock.Signature
      • NumpyBlock.from_column_data()
      • NumpyBlock.is_mmap
      • NumpyBlock.signature
  • meerkat.block.pandas_block module
    • PandasBlock
      • PandasBlock.Signature
      • PandasBlock.from_column_data()
      • PandasBlock.signature
  • meerkat.block.ref module
    • BlockRef
      • BlockRef.apply()
      • BlockRef.update()
      • BlockRef.block_indices
  • meerkat.block.tensor_block module
    • TensorBlock
      • TensorBlock.Signature
      • TensorBlock.from_column_data()
      • TensorBlock.signature
• meerkat.cells package
  • Submodules
  • meerkat.cells.abstract module
    • AbstractCell
      • AbstractCell.get()
      • AbstractCell.loader()
      • AbstractCell.metadata
  • meerkat.cells.spacy module
    • LazySpacyCell
      • LazySpacyCell.default_loader()
      • LazySpacyCell.from_state()
      • LazySpacyCell.get()
      • LazySpacyCell.get_state()
    • SpacyCell
      • SpacyCell.default_loader()
      • SpacyCell.from_state()
      • SpacyCell.get()
      • SpacyCell.get_state()
  • meerkat.cells.video module
    • VideoCell
      • VideoCell.get()
  • meerkat.cells.volume module
    • MedicalVolumeCell
      • MedicalVolumeCell.clear_metadata()
      • MedicalVolumeCell.default_loader()
      • MedicalVolumeCell.from_state()
      • MedicalVolumeCell.get()
      • MedicalVolumeCell.get_metadata()
      • MedicalVolumeCell.get_state()
• meerkat.columns package
  • Submodules
  • meerkat.columns.abstract module
    • AbstractColumn
      • AbstractColumn.append()
      • AbstractColumn.batch()
      • AbstractColumn.concat()
      • AbstractColumn.filter()
      • AbstractColumn.from_data()
      • AbstractColumn.full_length()
      • AbstractColumn.get_writer()
      • AbstractColumn.head()
      • AbstractColumn.is_equal()
      • AbstractColumn.streamlit()
      • AbstractColumn.tail()
      • AbstractColumn.to_pandas()
      • AbstractColumn.Columnable
      • AbstractColumn.data
      • AbstractColumn.formatter
      • AbstractColumn.is_mmap
      • AbstractColumn.logdir
      • AbstractColumn.metadata
  • meerkat.columns.arrow_column module
    • ArrowArrayColumn
      • ArrowArrayColumn.block_class
      • ArrowArrayColumn.concat()
      • ArrowArrayColumn.is_equal()
      • ArrowArrayColumn.to_numpy()
      • ArrowArrayColumn.to_pandas()
      • ArrowArrayColumn.to_tensor()
  • meerkat.columns.cell_column module
    • CellColumn
      • CellColumn.concat()
      • CellColumn.from_cells()
      • CellColumn.is_equal()
      • CellColumn.cells
  • meerkat.columns.image_column module
    • ImageColumn
      • ImageColumn.default_loader()
  • meerkat.columns.lambda_column module
    • LambdaCell
      • LambdaCell.get()
      • LambdaCell.data
    • LambdaColumn
      • LambdaColumn.concat()
      • LambdaColumn.fn()
      • LambdaColumn.is_equal()
  • meerkat.columns.list_column module
    • ListColumn
      • ListColumn.batch()
      • ListColumn.concat()
      • ListColumn.default_formatter()
      • ListColumn.from_list()
      • ListColumn.is_equal()
  • meerkat.columns.numpy_column module
    • NumpyArrayColumn
      • NumpyArrayColumn.block_class
      • NumpyArrayColumn.concat()
      • NumpyArrayColumn.from_array()
      • NumpyArrayColumn.from_npy()
      • NumpyArrayColumn.get_writer()
      • NumpyArrayColumn.is_equal()
      • NumpyArrayColumn.to_pandas()
      • NumpyArrayColumn.to_tensor()
      • NumpyArrayColumn.is_mmap
    • getattr_decorator()
  • meerkat.columns.pandas_column module
    • PandasSeriesColumn
      • PandasSeriesColumn.block_class
      • PandasSeriesColumn.cat
      • PandasSeriesColumn.dt
      • PandasSeriesColumn.str
      • PandasSeriesColumn.concat()
      • PandasSeriesColumn.from_array()
      • PandasSeriesColumn.is_equal()
      • PandasSeriesColumn.to_pandas()
      • PandasSeriesColumn.to_tensor()
    • getattr_decorator()
  • meerkat.columns.spacy_column module
    • SpacyColumn
      • SpacyColumn.from_docs()
      • SpacyColumn.from_texts()
      • SpacyColumn.read()
      • SpacyColumn.write()
      • SpacyColumn.docs
      • SpacyColumn.tokens
  • meerkat.columns.tensor_column module
    • TensorColumn
      • TensorColumn.block_class
      • TensorColumn.concat()
      • TensorColumn.from_data()
      • TensorColumn.get_writer()
      • TensorColumn.is_equal()
      • TensorColumn.to_pandas()
      • TensorColumn.to_tensor()
    • getattr_decorator()
  • meerkat.columns.video_column module
    • VideoColumn
      • VideoColumn.from_filepaths()
  • meerkat.columns.volume_column module
    • MedicalVolumeColumn
      • MedicalVolumeColumn.from_filepaths()
• meerkat.contrib package
  • Subpackages
    • meerkat.contrib.eeg package
      • Submodules
      • meerkat.contrib.eeg.data_utils module
    • meerkat.contrib.gqa package
      • build_gqa_dps()
      • crop_object()
      • read_gqa_dps()
      • write_gqa_dps()
    • meerkat.contrib.mimic package
      • Submodules
      • meerkat.contrib.mimic.gcs module
      • meerkat.contrib.mimic.modules module
      • meerkat.contrib.mimic.reports module
    • meerkat.contrib.torchvision package
      • get_cifar10()
      • get_torchvision_dataset()
    • meerkat.contrib.video_corruptions package
      • Submodules
      • meerkat.contrib.video_corruptions.transforms module
      • meerkat.contrib.video_corruptions.utils module
    • meerkat.contrib.visual_genome package
      • build_visual_genome_dps()
      • read_visual_genome_dps()
      • write_visual_genome_dps()
    • meerkat.contrib.wilds package
      • WILDSInputColumn
      • get_wilds_datapanel()
      • Submodules
      • meerkat.contrib.wilds.config module
      • meerkat.contrib.wilds.transforms module
  • Submodules
  • meerkat.contrib.celeba module
    • build_celeba_df()
    • download_celeba()
    • get_celeba()
  • meerkat.contrib.dew module
    • build_dew_dp()
  • meerkat.contrib.imagenet module
    • build_imagenet_dps()
  • meerkat.contrib.imagenette module
    • build_imagenette_dp()
    • download_imagenette()
  • meerkat.contrib.registry module
    • Registry
      • Registry.get()
      • Registry.register()
      • Registry.catalog
      • Registry.names
    • celeba()
    • cifar10()
    • dew()
    • enron()
    • imagenet()
    • imagenette()
    • inaturalist()
    • waterbirds()
    • yesno()
  • meerkat.contrib.siim_cxr module
    • cxr_transform()
    • cxr_transform_pil()
    • download_siim_cxr()
  • meerkat.contrib.visual_genome module
    • build_visual_genome_dps()
    • read_visual_genome_dps()
    • write_visual_genome_dps()
• meerkat.logging package
  • Submodules
  • meerkat.logging.utils module
    • initialize_logging()
    • set_logging_level()
    • set_logging_level_for_imports()
• meerkat.mixins package
  • Submodules
  • meerkat.mixins.blockable module
    • BlockableMixin
      • BlockableMixin.is_blockable()
      • BlockableMixin.block_class
  • meerkat.mixins.cloneable module
    • CloneableMixin
      • CloneableMixin.copy()
      • CloneableMixin.view()
    • StateClass
      • StateClass.klass
      • StateClass.state
  • meerkat.mixins.collate module
    • CollateMixin
      • CollateMixin.collate()
      • CollateMixin.collate_fn
    • identity_collate()
  • meerkat.mixins.file module
    • FileMixin
    • PathsMixin
  • meerkat.mixins.inspect_fn module
    • FunctionInspectorMixin
  • meerkat.mixins.io module
    • ColumnIOMixin
      • ColumnIOMixin.read()
      • ColumnIOMixin.write()
  • meerkat.mixins.lambdable module
    • LambdaMixin
      • LambdaMixin.to_lambda()
  • meerkat.mixins.mapping module
    • MappableMixin
      • MappableMixin.map()
  • meerkat.mixins.materialize module
    • MaterializationMixin
      • MaterializationMixin.lz
• meerkat.ml package
  • Submodules
  • meerkat.ml.activation module
    • ActivationExtractor
      • ActivationExtractor.add_hook()
    • ActivationOp
  • meerkat.ml.callbacks module
  • meerkat.ml.embedding_column module
    • EmbeddingColumn
      • EmbeddingColumn.build_faiss_index()
      • EmbeddingColumn.pca()
      • EmbeddingColumn.search()
      • EmbeddingColumn.umap()
      • EmbeddingColumn.visualize_umap()
  • meerkat.ml.huggingfacemodel module
  • meerkat.ml.instances_column module
  • meerkat.ml.metrics module
    • accuracy()
    • class_distribution()
    • compute_metric()
    • dice()
    • f1()
    • f1_macro()
    • f1_micro()
    • format_summary()
    • get_metric()
    • iou_score()
  • meerkat.ml.model module
    • Model
      • Model.activation()
      • Model.classification()
      • Model.evaluate()
      • Model.forward()
      • Model.remap_labels()
      • Model.training
  • meerkat.ml.prediction_column module
    • ClassificationOutputColumn
      • ClassificationOutputColumn.bincount()
      • ClassificationOutputColumn.entropy()
      • ClassificationOutputColumn.logits()
      • ClassificationOutputColumn.mode()
      • ClassificationOutputColumn.predictions()
      • ClassificationOutputColumn.preds()
      • ClassificationOutputColumn.probabilities()
      • ClassificationOutputColumn.probs()
  • meerkat.ml.segmentation_column module
  • meerkat.ml.tensormodel module
• meerkat.ops package
  • Submodules
  • meerkat.ops.concat module
    • concat()
  • meerkat.ops.merge module
    • merge()
• meerkat.pipelines package
  • Submodules
  • meerkat.pipelines.entitydatapanel module
• meerkat.tools package
  • Submodules
  • meerkat.tools.lazy_loader module
    • LazyLoader
  • meerkat.tools.utils module
    • MeerkatLoader
      • MeerkatLoader.find_python_module()
      • MeerkatLoader.find_python_name()
    • convert_to_batch_column_fn()
    • convert_to_batch_fn()
• meerkat.writers package
  • Submodules
  • meerkat.writers.abstract module
    • AbstractWriter
      • AbstractWriter.close()
      • AbstractWriter.finalize()
      • AbstractWriter.flush()
      • AbstractWriter.open()
      • AbstractWriter.write()
  • meerkat.writers.concat_writer module
    • ConcatWriter
      • ConcatWriter.close()
      • ConcatWriter.finalize()
      • ConcatWriter.flush()
      • ConcatWriter.open()
      • ConcatWriter.write()
  • meerkat.writers.numpy_writer module
    • NumpyMemmapWriter
      • NumpyMemmapWriter.close()
      • NumpyMemmapWriter.finalize()
      • NumpyMemmapWriter.flush()
      • NumpyMemmapWriter.open()
      • NumpyMemmapWriter.write()

Submodules

meerkat.config module

class ContribOptions

Bases: object

download_dir: str = '/home/docs/.meerkat/datasets'

class DisplayOptions

Bases: object

max_image_height: int = 128

max_image_width: int = 128

max_rows: int = 10

show_audio: bool = True

show_images: bool = True

meerkat.datapanel module

DataPanel class.

class DataPanel(data: dict | list | Dataset | None = None, *args, **kwargs)

Bases: CloneableMixin, FunctionInspectorMixin, LambdaMixin, MappableMixin, MaterializationMixin, ProvenanceMixin

Meerkat DataPanel class.

add_column(name: str, data: Sequence | ndarray | Series | Tensor, overwrite=False) → None

Add a column to the DataPanel.

append(dp: DataPanel, axis: str | int = 'rows', suffixes: Tuple[str] = None, overwrite: bool = False) → DataPanel

Append a batch of data to the dataset.

example_or_batch must have the same columns as the dataset (regardless of what columns are visible).

batch(batch_size: int = 1, drop_last_batch: bool = False, num_workers: int = 0, materialize: bool = True, shuffle: bool = False, *args, **kwargs)

Batch the dataset. TODO:

Parameters:

• batch_size – integer batch size

• drop_last_batch – drop the last batch if its smaller than batch_size

Returns:

batches of data

consolidate()

filter(function: Callable | None = None, with_indices=False, input_columns: str | List[str] | None = None, is_batched_fn: bool = False, batch_size: int | None = 1, drop_last_batch: bool = False, num_workers: int = 0, materialize: bool = True, pbar: bool = False, **kwargs) → DataPanel | None

Filter operation on the DataPanel.

classmethod from_arrow(table: Table)

Create a Dataset from a pandas DataFrame.

classmethod from_batch(batch: Dict[str, List | AbstractColumn]) → DataPanel

Convert a batch to a Dataset.

classmethod from_batches(batches: Sequence[Dict[str, List | AbstractColumn]]) → DataPanel

Convert a list of batches to a dataset.

classmethod from_csv(filepath: str, *args, **kwargs)

Create a Dataset from a csv file.

Parameters:

• filepath (str) – The file path or buffer to load from. Same as pandas.read_csv().

• *args – Argument list for pandas.read_csv().

• **kwargs – Keyword arguments for pandas.read_csv().

Returns:

The constructed datapanel.

Return type:

DataPanel

classmethod from_dict(d: Dict) → DataPanel

Convert a dictionary to a dataset.

Alias for Dataset.from_batch(..).

classmethod from_feather(path: str)

Create a Dataset from a feather file.

classmethod from_huggingface(*args, **kwargs)

Load a Huggingface dataset as a DataPanel.

Use this to replace datasets.load_dataset, so

>>> dict_of_datasets = datasets.load_dataset('boolq')

becomes

>>> dict_of_datapanels = DataPanel.from_huggingface('boolq')

classmethod from_jsonl(json_path: str) → DataPanel

Load a dataset from a .jsonl file on disk, where each line of the json file consists of a single example.

classmethod from_pandas(df: DataFrame)

Create a Dataset from a pandas DataFrame.

head(n: int = 5) → DataPanel

Get the first n examples of the DataPanel.

items()

keys()

map(function: Callable | None = None, with_indices: bool = False, input_columns: str | List[str] | None = None, is_batched_fn: bool = False, batch_size: int | None = 1, drop_last_batch: bool = False, num_workers: int = 0, output_type: type | Dict[str, type] = None, mmap: bool = False, mmap_path: str = None, materialize: bool = True, pbar: bool = False, **kwargs) → Dict | List | AbstractColumn | None

merge(right: DataPanel, how: str = 'inner', on: str | List[str] | None = None, left_on: str | List[str] | None = None, right_on: str | List[str] | None = None, sort: bool = False, suffixes: Sequence[str] = ('_x', '_y'), validate=None)

classmethod read(path: str, *args, **kwargs) → DataPanel

Load a DataPanel stored on disk.

remove_column(column: str) → None

Remove a column from the dataset.

streamlit()

tail(n: int = 5) → DataPanel

Get the last n examples of the DataPanel.

to_jsonl(path: str) → None

Save a Dataset to a jsonl file.

to_pandas() → DataFrame

Convert a Dataset to a pandas DataFrame.

update(function: Callable | None = None, with_indices: bool = False, input_columns: str | List[str] | None = None, is_batched_fn: bool = False, batch_size: int | None = 1, remove_columns: List[str] | None = None, num_workers: int = 0, output_type: type | Dict[str, type] = None, mmap: bool = False, mmap_path: str = None, materialize: bool = True, pbar: bool = False, **kwargs) → DataPanel

Update the columns of the dataset.

values()

write(path: str) → None

Save a DataPanel to disk.

property columns

Column names in the DataPanel.

property data: BlockManager

Get the underlying data (excluding invisible rows).

To access underlying data with invisible rows, use _data.

logdir: Path = PosixPath('/home/docs/meerkat')

property ncols

Number of rows in the DataPanel.

property nrows

Number of rows in the DataPanel.

property shape

Shape of the DataPanel (num_rows, num_columns).

meerkat.errors module

exception ConcatError

Bases: ValueError

exception ConcatWarning

Bases: RuntimeWarning

exception ConsolidationError

Bases: ValueError

exception ExperimentalWarning

Bases: FutureWarning

exception ImmutableError

Bases: ValueError

exception MergeError

Bases: ValueError

meerkat.provenance module

class ProvenanceMixin(*args, **kwargs)

Bases: object

get_provenance(include_columns: bool = False, last_parent_only: bool = False)

property node

class ProvenanceNode

Bases: object

add_child(node: ProvenanceNode, key: Tuple)

add_parent(node: ProvenanceNode, key: Tuple)

cache_repr()

get_provenance(last_parent_only: bool = False)

property children

property last_parent

property parents

class ProvenanceObjNode(obj: ProvenanceMixin)

Bases: ProvenanceNode

class ProvenanceOpNode(fn: callable, inputs: dict, outputs: object, captured_args: dict)

Bases: ProvenanceNode

class provenance(enabled: bool = True)

Bases: object

capture_provenance(capture_args: Sequence[str] | None = None)

get_nested_objs(data)

Recursively get DataPanels and Columns from nested collections.

is_provenance_enabled()

set_provenance(enabled=True)

visualize_provenance(obj: ProvenanceObjNode | ProvenanceOpNode, show_columns: bool = False, last_parent_only: bool = False)

meerkat.version module