Tag Archives: Matlab

[Matlab Trick] Squeeze

The Problem: I was given a three dimensional matrix (1xmxp), I need to convert it to a two dimensional matrix of size pxm.
The Trivial Solution: Nested loop. In addition to the time complexity, the solution make me feel like a dummy.

What I did instead: After 5 mins of google search, I found out a couple of solutions that I found helpful. I will share these with you below.

Let me explain with a concrete example. We need to generate a three dimensional matrix first, say size 1x3x4.

mat(:,:,1)=[1 2 3];
mat(:,:,2)=[4 5 6];
mat(:,:,3)=[7 8 9];
mat(:,:,4)=[10 11 12];

The goal is to convert this into a two dimensional matrix of size 4×3. The new matrix should look like the following:

new_mat =
[1 2 3;
4 5 6;
7 8 9;
10 11 12;]

Solution 1: Squeeze.

t = squeeze(mat) 
% the squeeze command removes the singleton dimensions
% a singleton dimension is a dimension of size 1 

t is a 3×4 matrix shown below:

t =
[1 4 7 10;
 2 5 8 11;
 3 6 9 12;]

Now we simply need to transpose it to get to our desired matrix.

new_mat = transpose(t)

One thing worth-noticing about the squeeze command: it simply removes the singleton dimensions. Applying squeeze function to matrix of size 1x3x4 (mat1 in the example below), or size 3x4x1 (mat2 in the example below), or size 3x1x4 will all result in matrix 3×4. This is illustrated below:

mat1(:,:,1)=[1 2 3];
mat1(:,:,2)=[4 5 6];
mat1(:,:,3)=[7 8 9];
mat1(:,:,4)=[10 11 12];

mat2(:,:,1)=[1; 2; 3];
mat2(:,:,2)=[4; 5; 6];
mat2(:,:,3)=[7; 8; 9];
mat2(:,:,4)=[10; 11; 12];

t1 = squeeze(mat1);
t2 = squeeze(mat2);

t1 and t2 are both 3×4, and they are exactly the same. I don’t know about everyone else, but I did not expect this to begin with.

Solution 2: Permute and Reshape
I will talk about this solution in my next blog post! see you soonish!

[Matlab] Import big file

The problem: I have this big txt file (well, relatively big, a little more than 200Mb) that I need to import into matlab and process.This file contain a table with characters and integers. The first row of the table is the column header. For my purpose, I only need data from specific columns (about 20 columns).

My initial solution: Since the file is not super huge and I do have a pretty decent machine. I just went with the importdata function. since it handles delimited txt file that contain string and integers pretty well in general.

The result: It took a long time, probably about 20 mins. And the file was not imported completely. The Header row was correctly imported, but only 4 rows of data were imported (I am still confused about why this is happening, and why Matlab did not even throw an error). But, I did learn something useful. It turned out that I have about 7000000 columns (according to the size of my header row).

My thought: I am sure you are thinking about the same thing as well, If I only need about 20 columns, it does not make much sense to read the whole file in. And apparently, reading the whole file in (at least using importdata function) has failed me.

The new plan:
(1) import the header, figure out which columns need to be imported
(2) import the content of the table line by line, only store the columns needed.

The code:
Alright, so here is how I did it. if you want to try this yourself, you can copy (start copying from the line that starts with YKZ) the following sample data into a text editor and save a txt file (in my example code, the sample data is saved in a txt file named text_file.txt) and see how it goes.

=============== [Sample data (space delimited)] ===============

YKZ Timestamp Temp Humidity Wind Weather
06-Sep-2013 01:00:00 6.6 89 4 clear
06-Sep-2013 05:00:00 5.9 95 1 clear
06-Sep-2013 09:00:00 15.6 51 5 mainly_clear
06-Sep-2013 13:00:00 19.6 37 10 mainly_clear
06-Sep-2013 17:00:00 22.4 41 9 mostly_cloudy
06-Sep-2013 21:00:00 17.3 67 7 mainly_clear
09-Sep-2013 01:00:00 15.2 91 8 clear
09-Sep-2013 05:00:00 19.1 94 7 n/a

==== [code (copy the code below to your matlab editor and run it directly) ] ======

% (1) import the header, figure out which columns need to be imported
clear all;
selected_col = {‘YKZ’,’Temp’,’Wind’};% specify columns need to be extracted by column names
fid = fopen(‘text_file.txt’);% create a file identifier. This does not load the content of the file
tline = fgetl(fid);% fgetl return the next line of the file identified by the fid. This function return a character string.
header = strsplit(tline,’ ‘); % now I parse the returned string using function strsplit, as a result I get the row header of the table. the first argument of strsplit is the string to be parsed, the second argument is the delimiter. Notice that the delimiter is optional, if a delimiter is not specified it uses white spaces.
idx = ismember(header,selected_col); % now we create a logical array to indicate the columns to be extracted.

% (2) import the content of the table line by line, only store the columns needed.
tdata={}; % initiate variable to store data
while ~feof(fid) % do the following step until end-of-file is reached
    tline = fgetl(fid); % every time fgetl is called, it reads in a new line from the file.
    delim_line = strsplit(tline,’ ‘); % parse the current line
    tdata(end+1,:)=delim_line(idx); % select data in columns specified by logical variable idx
end
fclose(fid); % close the original file

% (3) clean up
% Now, the extracted data are stored in cell array tdata (8×3 cell array)
% If you type tdata{1,2} in your command window, you should get:
ans =
6.6
% It is really important to recognize that 6.6 is not a number here.
% Now type class(tdata{1,2}) in your command window, you get:
ans =
char
% 6.6 here is of type char, so are all the other “numbers” in variable tdata. It is quite easy to convert them to numbers. Just one line of code.
num_data=cell2mat(cellfun(@(s) str2double(s), tdata(:,2:end), ‘UniformOutput’,false));
% the cellfun function apply str2double function to every cell in tdata(:,2:end) which convert them to type double, and function cell2mat convert the resulting cell array to matrix.

% (4) ta da, here is the data
col_header = header(idx); % column header
row_header = tdata(:,1); % row header
data = num_data; % here is the table content
clearvars -except col_header row_header data % clear all other intermediate variables

I hope you find this post helpful and feel free to leave a comment :)