In this case typically we hace more than one label per object you can see

Fp16 to fp32 Shit?

Its little bit rounding somewhat uncertaininty in gradient

Computation Linear Algebra cours

PASCAL data set with csv file for labeling and column specify if its validation

pd is for panda

iloc integer location [:,0] -- print all columns

a same as

from fastai.vision.all import *
path = untar_data(URLs.PASCAL_2007)

This dataset is different from the ones we have seen before, in that it is not structured by filename or folder but instead comes with a CSV (comma-separated values) file telling us what labels to use for each image. We can inspect the CSV file by reading it into a Pandas DataFrame:

It will also tell about validation

dataframe is table containing rows and columns

pd.head see first fews

df = pd.read_csv(path/'train.csv')
df.head()

Data Frame Methods

iloc

Data frame has methods index location

[ row index , Column index]

if there is bunch of , coluns you can get rid of that

you can pass column name

Python Data Analysis Higly recommended

Row index

df.iloc[0,:]

# ": " is an option

df.iloc[0]

fname       000005.jpg
labels           chair
is_valid          True
Name: 0, dtype: object

Column index

two options

indexing
Passing column name in data frame

df.iloc[:,0]

0       000005.jpg
1       000007.jpg
2       000009.jpg
3       000012.jpg
4       000016.jpg
           ...    
5006    009954.jpg
5007    009955.jpg
5008    009958.jpg
5009    009959.jpg
5010    009961.jpg
Name: fname, Length: 5011, dtype: object

df["fname"]

0       000005.jpg
1       000007.jpg
2       000009.jpg
3       000012.jpg
4       000016.jpg
           ...    
5006    009954.jpg
5007    009955.jpg
5008    009958.jpg
5009    009959.jpg
5010    009961.jpg
Name: fname, Length: 5011, dtype: object

Creating Column

tmp_df = pd.DataFrame({"a":[1,2],"b":[3,4]})

Adding Column

tmp_df['c'] = tmp_df['a']+ tmp_df['b']

Book by Creator

Python for data analysis

Constructing Data

We talked about datablock api its agreat way to create dataloaders

Lets do it from data from

Dataset vs Dataloader vs list Zip

Dataset

dataset is an abstract idea of class you can index into it and get labels have lenght of it

for example below is qualified

this is tuple containes

a = list(enumerate(string.ascii_lowercase))
a[0], len(a)

((0, 'a'), 26)

Dataloaders

When we have dataset and pass to dataloaders with particular batch size

it will create mini batch

it has bunch of dependent and independent elements

Lets see what zip does its

dl_a = DataLoader(a, batch_size =8,shuffle=True)

first(dl_a)

(tensor([17, 18, 10, 22,  8, 14, 20, 15]),
 ('r', 's', 'k', 'w', 'i', 'o', 'u', 'p'))

List zip

takes one of one or more elements of dependent and independent variable and print out

list(zip(first(dl_a)))

[(tensor([14, 19, 24,  3,  8,  2, 12, 16]),),
 (('o', 't', 'y', 'd', 'i', 'c', 'm', 'q'),)]

b=first(dl_a)

list(zip(b[0],b[1]))

[(tensor(12), 'm'),
 (tensor(9), 'j'),
 (tensor(20), 'u'),
 (tensor(5), 'f'),
 (tensor(25), 'z'),
 (tensor(1), 'b'),
 (tensor(19), 't'),
 (tensor(13), 'n')]

Python shortCut : zip

since we passing all of elements of variable

Star(variable) does the same insert into parameters

What is Dataset

list(zip(*b))

[(tensor(12), 'm'),
 (tensor(9), 'j'),
 (tensor(20), 'u'),
 (tensor(5), 'f'),
 (tensor(25), 'z'),
 (tensor(1), 'b'),
 (tensor(19), 't'),
 (tensor(13), 'n')]

Datasets

its an object with training and validation dataset

normally you start with file name

calculate , transform your file name into image and looking file name assign labels

Normally you do not start with tuple

if we pass list to dataset it will return dependent variable

a = list(string.ascii_lowercase)
a[0],len(a)

('a', 26)

it will return us tuple

(x,y) it is format

dss = Datasets(a)
dss[0]

('a',)

Take this and computer dependent and independt variable

here is functions we use to computer independednt variable a on the and dependent with b on the end

def f1(o): return o+"a"
def f2(o): return o+"b"

Transformation with Datasets

We can pass lit of transformation to do

dss = Datasets(a, [[f1]])
dss[0]

('aa',)

dss = Datasets(a, [[f1,f2]])

dss[0]

('aab',)

List Contaning F1 and F2

it will take a pass to f1 and also f2

this way build up independent and depepndent varaiables in fastai

Amazing

For example we passed f1 as image path dependent variable and independt as filename as label

dss = Datasets(a,[[f1],[f2]])

dss[0]

('aa', 'ab')

This way create dataloaders object

from datasets by passing datasets and batch size

dls = DataLoaders.from_dsets(dss,batch_size =4)
first(dls.train)

(('ta', 'oa', 'aa', 'ya'), ('tb', 'ob', 'ab', 'yb'))

DataBlock

Lets create empty datablock

its take our dataframe which we remember

we pass that

we see

datablock.datasets has created train and validation data for us

dblock = DataBlock()

dsets = dblock.datasets(df)

len(dsets.train),len(dsets.valid)

(4009, 1002)

first row is shuffled by default datablock assumed we have two things input and target train and valid =20 percent

x,y = dsets.train[0]
x,y

(fname       005239.jpg
 labels          person
 is_valid          True
 Name: 2621, dtype: object,
 fname       005239.jpg
 labels          person
 is_valid          True
 Name: 2621, dtype: object)

independent varable and dependent check

x["fname"]

'005239.jpg'

Lamba

its a function without name in our case we are saying lambda r take parameter r its gonna return fname column

Summary : What I learned about Datablock

Configure your dataset methods label and traning etc
call dataset function that will handle it

what if i will call dataloaders

Recall in image case we did

pets = Datablock (specify everything how labels to get from filename ,get_y etc...)

pets.dataloaders(image path)

dblock = DataBlock(get_x = lambda r: r["fname"],get_y = lambda r: r["labels"])
dsets = dblock.datasets(df)

dsets.train[0]

('006028.jpg', 'bicycle person')

Same Thing Again With Functions

Because randomly picking differnt dataset

def get_x(r): return r['fname']
def get_y(r): return r['labels']
dblock = DataBlock(get_x = get_x, get_y = get_y)
dsets = dblock.datasets(df)
dsets.train[0]

('007385.jpg', 'sofa')

Note

be careful about lambda if you want to save data block to use later

python dont support for lambda

Path.BASE_PATH = path

Lets Give the path of image

Now I have images as filename here from dataframe

I want to give a proper path to it

and

MultiCategory Label

def get_x(r): return path/'train'/r['fname']
def get_y(r): return r['labels'].split(' ')
dblock = DataBlock(get_x = get_x, get_y = get_y)
dsets = dblock.datasets(df)
dsets.train[0]

(Path('train/001930.jpg'), ['train', 'person'])

Results : Datablock

Independent variable = image dependendt are labels long list of zero and 1s

Its one hot encoding

0 -- no object 1- object present

dblock = DataBlock(blocks =(ImageBlock,MultiCategoryBlock),
                  get_x=get_x,get_y=get_y)
dsets =dblock.datasets(df)
dsets.train[0]

(PILImage mode=RGB size=500x375,
 TensorMultiCategory([0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.]))

	fname	labels	is_valid
0	000005.jpg	chair	True
1	000007.jpg	car	True
2	000009.jpg	horse person	True
3	000012.jpg	car	False
4	000016.jpg	bicycle	True