俄罗斯色情 - https://antabusetab.com. gemicai package Gemicai 050 documentation gemicai.Classifier module¶ This module contains a Classifier class which simplifies model training and evaluation process by abstracting away many implementation details. As a result this module allows the user to save a lot of time by providing a default implementation for many of the PyTorch options. Bases: object This class does all of the heavy lifting when it comes down to the model training, evaluation and tensor classification. During creation of this class it is possible to specify the following attributes: module (nn.Module) – specifies a model to train, for more information about models themselves please refer to the https://pytorch.org/docs/stable/torchvision/models classes (list) – a list of classes present in the dataset, this will be used in order to modify model’s last layer. For more information about how to obtain such a list please refer to the classes method of the gemicai.data_iterators.DicomoDataset layer_config (Optional[gemicai.classifier_functors.GEMICAIABCFunctor]) – optional parameter containing a functor that can be used to modify a given model. For more information please refer to the gemicai.classifier_functors module loss_function (Optional[nn.Module]) – optional parameter containing a loss function used during a training. optimizer (Optional[torch.optim.Optimizer]) – optional parameter containing an optimizer used during a training. enable_cuda (Optional[bool]) – if set to True the training will be done on the gpu otherwise the model will be trained on the cpu. Please note that training on a gpu is substantially faster. cuda_device (Optional[int]) – allows for selection of a particular cuda device if enable_cuda is set to True. PyTorch’s Device ids start at 0 and are incremented by one. RuntimeError – raised if the cuda device was selected but its not supported by the underlying machine TypeError – raised if any of the parameters is of an invalid type Takes in a tensor object and returns a list of predicted class types along with their certainties. tensor (torch.Tensor) – tensor to classify list of predicted classes and their certainty TypeError – raised if tensor does not have a torch.Tensor type Used to evaluate the model’s performance on a provided dataset. dataset (gemicai.data_iterators.GemicaiDataset) – dataset iterator used in order to evaluate a model’s performance. batch_size (int) – number (non-negative) of DataObject which will be feed into a classifier at once num_workers (int) – number (non-negative) of worker threads used to load data from the dataset pin_memory (bool) – whenever memory pages should be pinned or not. If set to false there is a possibility that memory pages might be moved to a swap decreasing overall program’s performance. verbosity (int) – specifies verbosity (non-negative) of training/evaluation output. 0 - no output, 1 - basic output, 2 or more - extended output output_policy (gemicai.output_policies.OutputPolicy) – specifies how and where to write the evaluation statistics tuple of model’s accuracy, number of total images and number of correctly classified images TypeError – if passed arguments are not of a correct type or their values are outside of valid bounds this method will raise a TypeError exception. ValueError – thrown whenever given data iterator object returns object containing more than two entries Used to load a Classifier object from a given file file_path (str) – a valid path to a file up to and including it’s extension type. zipped (bool) – whenever given file is zipped or not a valid Classifier object TypeError – thrown if the given path is of an invalid format Exception – thrown if Classifier object could not have been loaded in from the given file Saves current classifier object to the file system, it can be loaded back in using the gemicai.Classifier.Classifier.from_file method. file_path (str) – a valid path to a file, it does not require a file to exist. Optionally .gemclas file extension can be appended to a file path like so /home/test/classifier.gemclas, if the extension is not present it will be added automatically. zipped (bool) – whenever this object should be zipped or not TypeError – file_path is not a str type Used in order to select a device on which model training will be done. enable_cuda (Optional[bool]) – if set to True the training will be done on the gpu otherwise the model will be trained on the cpu. Please note that training on a gpu is substantially faster. cuda_device (Optional[int]) – allows for selection of a particular cuda device if enable_cuda is set to True. PyTorch’s Device ids start at 0 and are incremented by one. RuntimeError – raised if the cuda device was selected but its not supported by the underlying machine TypeError – raised if any of the parameters has an invalid type Sets specified layers to be either trainable or not. layers (list) – list of tuples specifying which layers should be trainable or not, eg. [(‘fc’, True), …]. Where ‘fc’ is a layer name and True specifies that it should be trainable. Note that instead of a layer name it is possible to pass ‘all’ in its place which will set every layer in the model to the specified mode, eg. [(‘all’, False)] makes every layer untrainable. TypeError – thrown if layers parameter has a wrong type Used to train a model. dataset (gemicai.data_iterators.GemicaiDataset) – dataset iterator used in order to train a model batch_size (int) – number (non-negative) of DataObject which will be feed into a classifier at once epochs (int) – specifies how many training iterations (non-negative) to perform. One iteration goes over a whole dataset. num_workers (int) – number (non-negative) of worker threads used to load data from the dataset pin_memory (bool) – whenever memory pages should be pinned or not. If set to false there is a possibility that memory pages might be moved to a swap decreasing overall program’s performance. verbosity (int) – specifies verbosity (non-negative) of training/evaluation output. 0 - no output, 1 - basic output, 2 or more - extended output test_dataset (Union[None, gemicai.data_iterators.GemicaiDataset]) – optional parameter, if a test_dataset iterator is passed and verbosity is set to at least 2 it will be used in order to evaluate model’s performance after a training epoch. output_policy (gemicai.output_policies.OutputPolicy) – specifies how and where to write the training statistics TypeError – if passed arguments are not of a correct type or their values are outside of valid bounds this method will raise a TypeError exception. ValueError – thrown whenever given data iterator object returns object containing more than two entries Called internally in order to validate passed arguments to the train and evaluate methods. dataset (gemicai.data_iterators.GemicaiDataset) – dataset iterator used in order to train/evaluate a model batch_size (int) – number (non-negative) of DataObject which will be feed into a classifier at once num_workers (int) – number (non-negative) of worker threads used to load data from the dataset pin_memory (bool) – whenever memory pages should be pinned or not. If set to false there is a possibility that memory pages might be moved to a swap decreasing overall program’s performance. test_dataset (Union[None, gemicai.data_iterators.GemicaiDataset]) – optional parameter, validates whenever a test_dataset iterator passed to the train function is a valid gemicai object verbosity (int) – specifies verbosity (non-negative) of training/evaluation output. 0 - no output, 1 - basic output, 2 or more - extended output output_policy (gemicai.output_policies.OutputPolicy) – specifies how and where to write the training/evaluation output epochs (int) – specifies how many training iterations (non-negative) to perform. One iteration goes over a whole dataset. TypeError – if passed arguments are not of a correct type or their values are outside of valid bounds this method will raise a TypeError exception. ValueError – thrown whenever given data iterator object returns object containing more than two entries gemicai.ClassifierTree module¶ Bases: object Bases: object gemicai.classifier_functors module¶ This module contains functors that can be used to modify models. Object of this class can be passed as an optional argument to the gemicai.Classifier.Classifier constructor, see it’s layer_config parameter for more information. Bases: gemicai.classifier_functors.GEMICAIABCFunctor Gemicai’s default functor which modifies model’s final layer depending on the number of passed classes. It works with most of the torchvision models, for more information about models themselves please refer to the https://pytorch.org/docs/stable/torchvision/models Bases: abc.ABC Every custom functor should extend this abstract base class. gemicai.data_iterators module¶ This module contains data iterators which are used in order to traverse a dataset and retrieve a relevant information from the DataObjects it contains. Bases: gemicai.data_iterators.DicomoDataset This class server as a proxy for the underlying iterators when the iter() method is called it returns a PickledDicomoFilePool object which can be iterated over, this results in a class that supports a parallel data loading. It’s constructor takes in the following parameters: base_path (str) – a valid path to a folder containing a .gemset datasets labels (list) – labels specifying which DataObject values except for a tensor will be returned by the next() call transform (Optional[any torchvision.transforms]) – optional transforms to be applied on the tensor constraints (Optional[dict]) – optional constraints that the DataObject has to fulfil in order to be returned by the next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’] label_counter_type (gemicai.label_counter.GemicaiLabelCounter) – label counter used by the summarize method TypeError – raised if any of the parameters has an invalid type This iterator supports a parallelized resource loading. always returns True Plots one image per value type. label (str) – label according to which we will look for a unique values, eg, ‘Modality’ cmap (str) – color scheme TypeError – raised whenever label is not a str Returns a dataset subset using provided constraints constraints (dict) – optional constraints that the DataObject has to fulfil in order to be returned by the next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’] a valid gemicai.data_iterators.ConcurrentPickledDicomObjectTaskSplitter object TypeError – raised whenever constraints parameter is not a dict Returns or prints a summary of all the DataObject values in the dataset selected by the label. label (str) – field label which values to summarize, for example ‘BodyPartExamined’ or ‘Modality’ print_summary (bool) – whenever to print or return an instance of gemicai.label_counters.GemicaiLabelCounter object if print_summary is set to false a class that extends a gemicai.label_counters.GemicaiLabelCounter TypeError – raised whenever one of the parameter has an invalid type Bases: gemicai.data_iterators.GemicaiDataset Every provided non-abstract data iterator extends this class and calls it’s __init__ method with a following argument: label_counter_type (gemicai.label_counter.GemicaiLabelCounter) – label counter used by the summarize method TypeError – raised if any of the parameters has an invalid type Should return a boolean specifying whenever current iterator supports parallelized resource loading. Returns a list of all of the classes in the dataset. label (str) – label to summarize on list of possible label values present in the dataset Creates a data iterator from the supplied folder which should contain .gemset data sets folder_path (str) – a valid path to an existing folder which contains .gemset datasets labels (Optional[list]) – labels specifying which DataObject values except for a tensor will be returned by the next() call transform (Optional[any torchvision.transforms]) – optional transforms to be applied on the tensor constraints (Optional[dict]) – optional constraints that the DataObject has to fulfil in order to be returned by the next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’] a valid gemicai.data_iterators.ConcurrentPickledDicomObjectTaskSplitter object NotADirectoryError – raised whenever passed folder_path is invalid Creates a data iterator for a supplied .gemset file file_path (str) – a valid path to a .gemset file labels (Optional[list]) – labels specifying which DataObject values except for a tensor will be returned by the next() call transform (Optional[any torchvision.transforms]) – optional transforms to be applied on the tensor constraints (Optional[dict]) – optional constraints that the DataObject has to fulfil in order to be returned by the next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’] a valid gemicai.data_iterators.PickledDicomoDataSet object FileNotFoundError – raised whenever file_path does not point to any valid file Created a data iterator from the supplied file or folder path data_set_path – a valid path to an existing folder which contains .gemset datasets or a valid path to a .gemset file labels (Optional[list]) – labels specifying which DataObject values except for a tensor will be returned by the next() call constraints (Optional[dict]) – optional constraints that the DataObject has to fulfil in order to be returned by the next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’] gemicai.data_iterators.PickledDicomoDataSet object if file path was supplied otherwise gemicai.data_iterators.ConcurrentPickledDicomObjectTaskSplitter object FileNotFoundError – raised whenever file_path does not point to any valid file NotADirectoryError – raised whenever passed folder_path is invalid Plots one image per value type. label (str) – label according to which we will look for a unique values, eg, ‘Modality’ cmap (str) – color scheme TypeError – raised whenever label is not a str Should return a subset of a current dataset. constraints (dict) – dictionary with a dataset constraints, eg. ‘Modality’: ‘CT’ a correct user defined iterator type which extends gemicai.data_iterators.GemicaiDataset Returns or prints a summary of all the DataObject values in the dataset selected by the label. label (str) – field label which values to summarize, for example ‘BodyPartExamined’ or ‘Modality’ print_summary (bool) – whenever to print or return an instance of gemicai.label_counters.GemicaiLabelCounter object if print_summary is set to false a class that extends a gemicai.label_counters.GemicaiLabelCounter TypeError – raised whenever one of the parameter has an invalid type Bases: abc.ABC, torch.utils.data.dataset.IterableDataset This interface class serves as a basis for the every Gemicai’s data iterator. Should return a boolean specifying whenever current iterator supports parallelized resource loading. Should return a list of all the classes in the dataset. label (str) – label to summarize on list of possible label values present in the dataset Should plot one image per class. label (str) – label according to which we will look for a unique values cmap (str) – color scheme Should return a subset of a current dataset. constraints (dict) – dictionary with a dataset constraints, eg. ‘Modality’: ‘CT’ a correct user defined iterator type which extends gemicai.data_iterators.GemicaiDataset Should return or print a summary of all the DataObject values in the dataset selected by the label. label (str) – field label which values to summarize, for example ‘BodyPartExamined’ or ‘Modality’ print_summary (bool) – whenever to print or return an instance of gemicai.label_counters.GemicaiLabelCounter object if print_summary is set to false a class that extends a gemicai.label_counters.GemicaiLabelCounter Bases: gemicai.data_iterators.DicomoDataset This class takes in a path to a folder containing a .gemset datasets and iterates over them. It’s constructor takes in the following parameters: base_path (str) – a path to a valid folder containing a .gemset datasets labels (list) – labels specifying which DataObject values except for a tensor will be returned by the next() call transform (Optional[any torchvision.transforms]) – optional transforms to be applied on the tensor constraints (Optional[dict]) – optional constraints that the DataObject has to fulfil in order to be returned by the next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’] label_counter_type (gemicai.label_counter.GemicaiLabelCounter) – label counter used by the summarize method TypeError – raised if any of the parameters has an invalid type NotADirectoryError – raised if the passed path does not point to any directory This iterator does not support a parallelized resource loading. always returns False Returns a dataset subset using provided constraints constraints (dict) – optional constraints that the DataObject has to fulfil in order to be returned by the next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’] a valid gemicai.data_iterators.PickledDicomoDataFolder object TypeError – raised whenever constraints parameter is not a dict Bases: gemicai.data_iterators.DicomoDataset This class takes in a valid path to a .gemset dataset and iterates over it. It’s constructor takes in the following parameters: pickle_path (str) – a path to a valid .gemset file labels (Optional[list]) – labels specifying which DataObject values except for a tensor will be returned by the next() call transform (Optional[any torchvision.transforms]) – optional transforms to be applied on the tensor constraints (Optional[dict]) – optional constraints that the DataObject has to fulfil in order to be returned by the next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’] label_counter_type (gemicai.label_counter.GemicaiLabelCounter) – label counter used by the summarize method TypeError – raised if any of the parameters has an invalid type FileNotFoundError – raised whenever passed pickle_path does not point to any existing file This iterator does not support a parallelized resource loading. always returns False Returns a dataset subset using provided constraints constraints (dict) – optional constraints that the DataObject has to fulfil in order to be returned by the next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’] a valid gemicai.data_iterators.PickledDicomoDataSet object TypeError – raised whenever constraints parameter is not a dict Bases: gemicai.data_iterators.DicomoDataset This class takes in a list of files as an input and iterates over them. It’s constructor takes in the following parameters: file_pool (list) – list of a valid file paths to .gemset datasets labels (list) – labels specifying which DataObject values except for a tensor will be returned by the next() call transform (Optional[any torchvision.transforms]) – optional transforms to be applied on the tensor constraints (Optional[dict]) – optional constraints that the DataObject has to fulfil in order to be returned by the next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’] label_counter_type (gemicai.label_counter.GemicaiLabelCounter) – label counter used by the summarize method TypeError – raised if any of the parameters has an invalid type FileNotFoundError – raised if some path in the file_pool does not point to any existing file This iterator does not support a parallelized resource loading. always returns False Returns a dataset subset using provided constraints constraints (dict) – optional constraints that the DataObject has to fulfil in order to be returned by the next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’] a valid gemicai.data_iterators.PickledDicomoFilePool object TypeError – raised whenever constraints parameter is not a dict gemicai.data_objects module¶ This module contains data objects used by the Gemicai’s iterators Bases: abc.ABC Every custom data object should extend this abstract base class and call it’s constructor. This method should create and return a DataObject instance. filename (Union[os.path, str]) – path to a valid file name Call to this method should plot DataObject’s tensor to the screen. Bases: gemicai.data_objects.DataObject Gemicai’s default data object Creates a DicomoObject from a specified file. filename (Union[os.path, str]) – a valid dicom file path labels (list) – labels which values will be taken from the Dicom object. The pixel_array field should not be specified as it is one of the default fields fetched internally. tensor_size (Optional[tuple]) – used to resize a tensor. If left unspecified it will try to use the current image size otherwise it will use the specified values. Correct format ((int)x, (int)y) DicomoObject instance Returns a label value of a given label type. item (str) – string with a objects label value of a label or None if the object does not contain it Checks whenever the object meets a certain type of criteria. constraints (dict) – constraints to check against eg. ‘Modality’: ‘CT’ True if the object meets criteria, False otherwise Prints labels and plots the tensor. cmap (str) – color scheme gemicai.dicom_utilities module¶ This module contains some utility functions that are used by the Gemicai in order to interface with dicom objects Creates a Gemicai dataset from the data_origin (it should contain a valid dicom objects) and puts them in the data_destination. data_origin (Union[str, io.path]) – path to a folder containing dicom files data_destination (Union[str, io.path]) – path to a destination where the gemsets will be outputted relevant_labels (list) – specify which labels along with their values to extract from the dicom file and put into gemicai.data_objects.DicomObject, eg. [‘Modality’] in this case DicomObject will contain a tensor and its Modality field_values (Optional[list]) – dataset will contain only objects which fulfil specified critieria, eg. [(‘Modality’, [‘CT’, ‘MG’]), …] in this case dataset will contain only objects whose Modality is set to CT or MG objects_per_file (Optional[int]) – specifies how many objects one gemicai dataset should contain. A default value is 1000 pick_middle (bool) – specifies whenever instead of taking all images from the series only the middle one is taken. This can be useful if someone is dealing with series spanning a multiple of dicom objects. verbosity (int) – optional non-negative parameter, if set to one it will output how long it took to process all of the data from data_origin NotADirectoryError – raised if data_origin or data_destination does not point to an existing directory TypeError – raised if any of the parameters has a wrong type or its value is out of the accepted bounds Extracts an image from the dicom file and creates a tensor out of it ds (pydicom.dataset.FileDataset) – dicom object to extract an image from torch.Tensor Loads in a given dicom file using a pydicom library filename (Union[str, os.path]) – a path to the .dcm.gz or .dcm file pydicom.dataset.FileDataset or pydicom.dicomdir.DicomDir TypeError – raised if the file extension does not end with .dcm nor .gz Plots image stored in a given dicom file. If a path given instead it will try to load a specified file first. dcm (Union[str, pydicom.dataset.FileDataset]) – dicom object or a valid path to a dicom file cmap (str) – color scheme gemicai.label_counters module¶ This module contains label counters which are used by the data iterators in order to count distinct data classes present in the dataset Bases: abc.ABC Every custom label counter should extend this abstract base class This function is called whenever we have to count number of unique classes in a given input labels (any) – in case of a user defined label counter it has to hold values to check against Bases: gemicai.label_counters.GemicaiLabelCounter Gemicai’s default label counter implementation This function checks if a given input is in it’s internal mapping if not it is added to it and it’s counter is set to one, otherwise if it is already present then the counter is incremented by one. labels (Union[list, str, pydicom.valuerep.IS]) – contains labels to count gemicai.output_policies module¶ This module contains output policies. Such policy can be supplied as an optional parameter during model training or evaluation in order to log, save, or print statistics related to the model’s performance. Bases: abc.ABC Every custom policy should extend this abstract base class. Called after a model evaluation finishes if verbosity is set to 1. total (int) – number of total objects model was evaluated on correct (int) – number of correctly classified objects acc (float) – overall accuracy of the model Called after a model evaluation finishes if verbosity is equal or greater than 2. classes (list) – list with class names on which model was evaluated class_total (list) – list with a number of classes on which model was evaluated class_correct (list) – list with a number of properly classified classes This function is called every time an epoch ends, all of the important training statistics are taken as an input. epoch (int) – current training epoch epoch running_loss (float) – current total running loss of the model total (int) – number of the images that model has trained on train_acc (str) – models accuracy on the provided train dataset test_acc (str) – models accuracy on the optionally provided eval dataset elapsed (str) – total time it took to run the epoch eta (str) – an estimated time when training will end Called once when training has finished. start (datetime.datetime) – time when the training has started now (datetime.datetime) – current time Class implementing this method should make sure to specify how to handle a training header call. This call happens once before training. It’s purpose is to beautify training_epoch_stats output by providing some context into what categories of data the user is looking at. Bases: gemicai.output_policies.OutputPolicy This policy allows to output training statistics to the console Outputs model evaluation statistics to the console if verbosity is set to 1. total (int) – number of total objects model was evaluated on correct (int) – number of correctly classified objects acc (float) – overall accuracy of the model Outputs model evaluation statistics to the console if verbosity is equal or greater than 2. classes (list) – list with class names on which model was evaluated class_total (list) – list with a number of classes on which model was evaluated class_correct (list) – list with a number of properly classified classes an input and outputted to the console. epoch (int) – current training epoch epoch running_loss (float) – current total running loss of the model total (int) – number of the images that model has trained on train_acc (str) – models accuracy on the provided train dataset test_acc (str) – models accuracy on the optionally provided eval dataset elapsed (str) – total time it took to run the epoch eta (str) – an estimated time when training will end Outputs elapsed training time to the console. start (datetime.datetime) – time when the training has started now (datetime.datetime) – current time Prints a training header to the console Bases: gemicai.output_policies.ToConsole, gemicai.output_policies.ToExcelFile This output policy is a composition of ToConsole and ToExcelFile policies. Called after a model evaluation finishes if verbosity is set to 1. Outputs the training statistics to the console and to the specified excel file. total (int) – number of total objects model was evaluated on correct (int) – number of correctly classified objects acc (float) – overall accuracy of the model Called after a model evaluation finishes if verbosity is equal or greater than 2. Outputs the training statistics to the console and to the specified excel file. classes (list) – list with class names on which model was evaluated class_total (list) – list with a number of classes on which model was evaluated class_correct (list) – list with a number of properly classified classes as an input and outputs them to the console and the specified excel file. epoch (int) – current training epoch epoch running_loss (float) – current total running loss of the model total (int) – number of the images that model has trained on train_acc (str) – models accuracy on the provided train dataset test_acc (str) – models accuracy on the optionally provided eval dataset elapsed (str) – total time it took to run the epoch eta (str) – an estimated time when training will end Called once when training has finished. Outputs elapsed time to the console and the specified excel file. start (datetime.datetime) – time when the training has started now (datetime.datetime) – current time Outputs training header to the console and the specified excel file Bases: gemicai.output_policies.OutputPolicy This policy allows to output training statistics to the excel file Called after a model evaluation finishes if verbosity is set to 1. Outputs a training statistics to the specified excel file. total (int) – number of total objects model was evaluated on correct (int) – number of correctly classified objects acc (float) – overall accuracy of the model Called after a model evaluation finishes if verbosity is equal or greater than 2. Outputs a training statistics to the specified excel file. classes (list) – list with class names on which model was evaluated class_total (list) – list with a number of classes on which model was evaluated class_correct (list) – list with a number of properly classified classes This function writes each entry from data_list into a separate cell. data_list (list) – list with data to be written cells (list) – list with column names. Should have at least as many entries as data_list. an input and outputted to the specified excel file. epoch (int) – current training epoch epoch running_loss (float) – current total running loss of the model total (int) – number of the images that model has trained on train_acc (str) – models accuracy on the provided train dataset test_acc (str) – models accuracy on the optionally provided eval dataset elapsed (str) – total time it took to run the epoch eta (str) – an estimated time when training will end Called once when training has finished, it outputs the elapsed time to the specified excel file. start (datetime.datetime) – time when the training has started now (datetime.datetime) – current time Outputs a training header to the excel file