Real Software Forums

The forum for Real Studio and other Real Software products.
[ REAL Software Website | Board Index ]
It is currently Fri Dec 13, 2019 8:47 pm
xojo

All times are UTC - 5 hours




Post new topic Reply to topic  [ 39 posts ]  Go to page Previous  1, 2, 3  Next
Author Message
 Post subject: Re: MBS Plugins - Compression plugin
PostPosted: Wed Aug 24, 2011 2:55 pm 
Offline

Joined: Wed May 20, 2009 11:02 am
Posts: 423
timhare wrote:
Well, this isn't a project they're talking about, it's a separate component that you would include in a project, so they can't drop the dll into it. Just won't work.

Yes, I see it is a different approach, but finally it is the same idea, store a property in the application to create a file out of it. I would say drag and dropping the file in the application seems simpler, and if there isn't any drawback to that method I am not aware of (as compared to the method presented in this thread), then I would go for the drag and drop option.

So, my question was if there is any advantage to any of the methods. I was thinking maybe there could be a size difference in storing the data as a binary property or as a string?

timhare wrote:
In your sample code, you can just say
bs.write(ImportedData)

No need to loop over each byte.

Great, thank you.

Pixe

_________________
Using RS2011r4.3 on Windows7.


Top
 Profile  
Reply with quote  
 Post subject: Re: MBS Plugins - Compression plugin
PostPosted: Wed Aug 24, 2011 3:28 pm 
Offline

Joined: Fri Jan 06, 2006 3:21 pm
Posts: 12388
Location: Portland, OR USA
Drag/Drop is definitely the better option, but is not available in this case.


Top
 Profile  
Reply with quote  
 Post subject: Re: MBS Plugins - Compression plugin
PostPosted: Wed Aug 24, 2011 4:58 pm 
Offline

Joined: Wed May 20, 2009 11:02 am
Posts: 423
Uumm, there must be something I am missing. What do you mean "it is not available"?

Do you mean it cannot be done with a dll file? Can't it be done just with any kind of file?

I just tried it using a dll file and it does wrok. What am I missing then?

_________________
Using RS2011r4.3 on Windows7.


Top
 Profile  
Reply with quote  
 Post subject: Re: MBS Plugins - Compression plugin
PostPosted: Wed Aug 24, 2011 5:05 pm 
Offline

Joined: Fri Jan 06, 2006 3:21 pm
Posts: 12388
Location: Portland, OR USA
I mean, they are providing a component that you would use in your project. YOU could drag/drop a dll into your project, but their code has no way of doing that. The mechanism they're using allows them to include a dll in their component that then you include in your project.


Top
 Profile  
Reply with quote  
 Post subject: Re: MBS Plugins - Compression plugin
PostPosted: Wed Aug 24, 2011 5:27 pm 
Offline

Joined: Wed May 20, 2009 11:02 am
Posts: 423
OK. Thanks for the clarification.

Pixe.

_________________
Using RS2011r4.3 on Windows7.


Top
 Profile  
Reply with quote  
 Post subject: Re: MBS Plugins - Compression plugin
PostPosted: Thu Aug 25, 2011 3:13 am 
Offline

Joined: Wed May 20, 2009 11:02 am
Posts: 423
I performed a test to see the size difference of the built application using the two methods. I used a 324 KB dll file, on windows.

Size of the built application without any data: 4226 KB (it contains only a pushbutton and a few lines of code)
Size of the built application containing the drag and dropped dll file: 4499 KB (same as previous one, plus the dll file)
Size of the built application storing the data as strings: 5103 KB (same as previous one, this time using the string method)

It looks that storing the file as text can bloat it up, which should be taken into consideration for medium-large files (maybe not for small ones).

Pixe

_________________
Using RS2011r4.3 on Windows7.


Top
 Profile  
Reply with quote  
 Post subject: Re: MBS Plugins - Compression plugin
PostPosted: Thu Aug 25, 2011 3:42 am 
Offline

Joined: Fri Sep 30, 2005 10:01 am
Posts: 283
Location: Germany, Munich
The hex encoding used by Matthew is indeed a bit wasteful. A binhex encoding would probably be more efficient. Or your own encoding that uses more codes than what base64 uses (base64 only uses 64 out of 256 possible codes in a binary string, and RB's limis is what you can put into a String constants, which is probably 128-32 chars, since you can't use the upper 128 codes as RB stores them as UTF-8, which won't allow arbitrary combinations in that range).
Or you'd try to compress the base64 stream. Which might lead to a chicken&egg situation... ;)

_________________
User of RB since first version. Provider of many free and outdated plugins.
Code for sharing: http://www.tempel.org/RB/Resources
Arbed, a unique tool for editing projects: http://www.tempel.org/Arbed
Zip compression classes: http://www.tempel.org/RB/ZipPackage


Top
 Profile  
Reply with quote  
 Post subject: Re: MBS Plugins - Compression plugin
PostPosted: Thu Aug 25, 2011 8:57 pm 
Offline
User avatar

Joined: Sun Aug 12, 2007 10:10 am
Posts: 1086
Location: Boiling Springs, SC
tempel wrote:
The hex encoding used by Matthew is indeed a bit wasteful. A binhex encoding would probably be more efficient. Or your own encoding that uses more codes than what base64 uses (base64 only uses 64 out of 256 possible codes in a binary string, and RB's limis is what you can put into a String constants, which is probably 128-32 chars, since you can't use the upper 128 codes as RB stores them as UTF-8, which won't allow arbitrary combinations in that range).
Or you'd try to compress the base64 stream. Which might lead to a chicken&egg situation... ;)


Here is the code...how can it be improved. Would love to see ideas as this would be quite efficient if the size didn't get to large..possibly huffman compression before hexing? It would be great for packaging say..a Zip file into code..creating the zip on execution and unpacking files using your "plugin"? Hope the code helps someone and would love to see howto improve it...P...I'll send the other code to you as well tomorrow..this one is cross platform..the other is windows specific and uses copymem api :-)

Code:
Create Properties:

Res as String
ResourceFile As FolderItem
Resources As String


Methods:
Sub AddRes(strRes as String)
if Res= "" then
Res = strRes + EndOfLine
else
Res = Res + strRes + EndOfLine
end if
End Sub



Sub AddLog(strLog as String)
if Resources = "" then
Resources = strlog + EndOfLine
else
Resources = Resources + strLog + EndOfLine
end if
End Sub


Button:cmdSavePackage
Sub Action()
Dim y as FolderItem
dim x as TextOutputStream

y = GetSaveFolderItem(FileTypes1.Text, "Untitled-Package.txt")

x = TextOutputStream.Create(y)
x.Write(txtCode.text)
x.Close
msgbox "Package File Saved Successfully!"
End Sub


Button:cmdGetFile
Sub Action()
ResourceFile = GetOpenFolderItem("")
Label1.Text = ResourceFile.AbsolutePath
End Sub


Button:cmdPackage
Sub Action()

Dim MyResource as String
Dim ShortResourceFileName As String
Dim xResource as TextInputStream
Dim xResult as String
Dim WW as Variant, BCount as Integer, B as String, a as String, c as Integer, CC as string
Dim i as Double
Dim CD(16) as String


MyResource = Label1.text

If MyResource = "" Then
MsgBox "Please choose a file first!"
Exit Sub
End If

ShortResourceFileName = ResourceFile.DisplayName
xResource = TextInputStream.Open(ResourceFile)
xResult = xResource.ReadAll
xResource.close

WW = 1
BCount = 0

B = " ResourceData(" + Trim(WW) + ") = " + Chr(34)
For i = 1 To len(xResult)
a = mid(xResult,i,1)
c = Asc(a)
CC = Hex(c)
If Len(CC) <> 2 Then
CC = CC + " "
End If
BCount = BCount + 1
B = B + CC
If BCount = 240 Then
B = B + Chr(34)
AddRes B
WW = WW + 1
BCount = 0
B = " ResourceData(" + Trim(WW) + ") = " + Chr(34)
End If
Next i
B = B + Chr(34)
AddRes B

AddLog "Sub UnpackFile()"
CD(1) = " '============================================="
CD(2) = " ' PASTE THIS ENTIRE CODE INTO YOUR METHOD SECTION "+ endofline +_
" 'Generated By Simulanics REAL File Packager"
CD(3) = " '============================================="
For i = 1 To 3
AddLog CD(i)
Next i

CD(4) = "Dim ResourceData(" + Trim(WW) + ") as String" + EndOfLine +_
"Dim f as FolderItem" + EndOfLine +_
"Dim c,k, iLen As Integer" + EndOfLine +_
"Dim iByte As Integer" + EndOfLine +_
"Dim s As String" + EndOfLine +_
"Dim b as BinaryStream" + EndOfLine
AddLog CD(4)

if RadExist.Value = True then
Addlog "f = GetFolderItem(" + chr(34) + chr(34) + ").Child(" +chr(34) + ShortResourceFileName + chr(34) +")"
AddLog "If (f <> Nil) And f.Exists Then"
addlog "f.Delete"
addlog "End If"
end if

if radSkip.value = true then
Addlog "f = GetFolderItem(" + chr(34) + chr(34) + ").Child(" +chr(34) + ShortResourceFileName + chr(34) +")"
AddLog "If (f <> Nil) And f.Exists Then"
addlog "Exit Sub"
addlog "End If"
end if

addlog ""
AddLog Res
addlog ""

CD(5) = " If f <> Nil then"
CD(6) = " b = BinaryStream.Create(f, True)"
CD(7) = " For c = 1 To " + Trim(WW)
CD(8) = " iLen = ResourceData(c).Len"
CD(9) = " For k = 1 To iLen Step 2"
CD(10) = " s = Mid(ResourceData(c), k, 2)"
CD(11) = " iByte = Val("+ chr(34)+ "&h"+ chr(34) +" + s)"
CD(12) = " b.WriteByte(iByte)"
CD(13) = " Next k"
CD(14) = " Next c"
CD(15) = " b.Close"
CD(16) = " End if "
For i = 5 To 16
AddLog CD(i)
Next i
AddLog "End Sub "

txtCode.text = Resources

End Sub

_________________
Matthew A. Combatti
Real Studio 2012 r1.2

Visit Xojo Developer's Spot!
Systems I Use:
Windows XP/Windows Vista/Windows Server 2008 r2/Windows 7/Windows 8
Mac OSX 10.5/Mac OSX 10.6/Mac OSX Server/Ubuntu/Debian/Suse/Red Hat/
Windows Server 2011/CentOS 5.4 /ReactOS/SimOS

~All Xojo Compatible~


Top
 Profile  
Reply with quote  
 Post subject: Re: MBS Plugins - Compression plugin
PostPosted: Thu Aug 25, 2011 9:58 pm 
Offline
User avatar

Joined: Sun Aug 12, 2007 10:10 am
Posts: 1086
Location: Boiling Springs, SC
pixe656 wrote:
I performed a test to see the size difference of the built application using the two methods. I used a 324 KB dll file, on windows.

Size of the built application without any data: 4226 KB (it contains only a pushbutton and a few lines of code)
Size of the built application containing the drag and dropped dll file: 4499 KB (same as previous one, plus the dll file)
Size of the built application storing the data as strings: 5103 KB (same as previous one, this time using the string method)

It looks that storing the file as text can bloat it up, which should be taken into consideration for medium-large files (maybe not for small ones).

Pixe


Have you heard about UPX?
http://upx.sourceforge.net/

What are these sizes after dropping them on the UPX executable..usually...8MB becomes ~2MB thus becomming 20-25% their original size :-) this can be done for all windows exe/dll/ocx :-) just curious as to the size after being packed...and packing does not requiring unpacking nor degrade performance...just removes all the "unnecessary" disk space from op code :-)

_________________
Matthew A. Combatti
Real Studio 2012 r1.2

Visit Xojo Developer's Spot!
Systems I Use:
Windows XP/Windows Vista/Windows Server 2008 r2/Windows 7/Windows 8
Mac OSX 10.5/Mac OSX 10.6/Mac OSX Server/Ubuntu/Debian/Suse/Red Hat/
Windows Server 2011/CentOS 5.4 /ReactOS/SimOS

~All Xojo Compatible~


Top
 Profile  
Reply with quote  
 Post subject: Re: MBS Plugins - Compression plugin
PostPosted: Thu Aug 25, 2011 10:09 pm 
Offline
User avatar

Joined: Sun Aug 12, 2007 10:10 am
Posts: 1086
Location: Boiling Springs, SC
by the way how do you perform binhex in 2011? it use to be a command...no longer available...am curious as it did serve to be quite useful especially when performing CRC checksums :-)

_________________
Matthew A. Combatti
Real Studio 2012 r1.2

Visit Xojo Developer's Spot!
Systems I Use:
Windows XP/Windows Vista/Windows Server 2008 r2/Windows 7/Windows 8
Mac OSX 10.5/Mac OSX 10.6/Mac OSX Server/Ubuntu/Debian/Suse/Red Hat/
Windows Server 2011/CentOS 5.4 /ReactOS/SimOS

~All Xojo Compatible~


Top
 Profile  
Reply with quote  
 Post subject: Re: MBS Plugins - Compression plugin
PostPosted: Fri Aug 26, 2011 2:06 am 
Offline

Joined: Fri Sep 30, 2005 10:01 am
Posts: 283
Location: Germany, Munich
I have no easy answer to optimizing the code, you'll have to figure out the math yourself.

Binhex? Did I write binhex? If I did - I meant Base64. just try it out: use encodebase64 and decodebase64 to create the encoded strings instead of doing the hex-encoding you do now. i bet that'll compress the data better because you do not need to add a space between each byte as you do now. simply create base64 segments of 64 chars or something like that, line by line.

To make your decompression code much faster do this:
1. first create a memoryblock with the size of the resulting file
2. instead of writing each byte into the binaryfile, put them into the memoryblock
3. write the memoryblock to the file.

Also, it's not efficient to concatenate lots of strings to each other with "+". That gets slower and slower the more strings you add up that way. instead, define a array and append the single strings to that array. then, when done with all the appending, get the resulting added-up string by calling the join(array,"") function.

so much for simple optimizations.

_________________
User of RB since first version. Provider of many free and outdated plugins.
Code for sharing: http://www.tempel.org/RB/Resources
Arbed, a unique tool for editing projects: http://www.tempel.org/Arbed
Zip compression classes: http://www.tempel.org/RB/ZipPackage


Top
 Profile  
Reply with quote  
 Post subject: Re: MBS Plugins - Compression plugin
PostPosted: Fri Aug 26, 2011 5:13 pm 
Offline

Joined: Wed May 20, 2009 11:02 am
Posts: 423
simulanics wrote:
pixe656 wrote:
I performed a test to see the size difference of the built application using the two methods. I used a 324 KB dll file, on windows.

Size of the built application without any data: 4226 KB (it contains only a pushbutton and a few lines of code)
Size of the built application containing the drag and dropped dll file: 4499 KB (same as previous one, plus the dll file)
Size of the built application storing the data as strings: 5103 KB (same as previous one, this time using the string method)

It looks that storing the file as text can bloat it up, which should be taken into consideration for medium-large files (maybe not for small ones).

Pixe


Have you heard about UPX?
http://upx.sourceforge.net/

What are these sizes after dropping them on the UPX executable..usually...8MB becomes ~2MB thus becomming 20-25% their original size :-) this can be done for all windows exe/dll/ocx :-) just curious as to the size after being packed...and packing does not requiring unpacking nor degrade performance...just removes all the "unnecessary" disk space from op code :-)


I didn't know upx, just tried and worked fine, thank you!

I got a 30-35% size reduction for the files I mentioned before. The size reduction is not as impressive as what you said, but it might be because the RB framework is already quite compact? Still, I think this is a great tool.

I have been thinking about this topic, and had the idea of creating an in-memory (RAM) dll file. This way there would not be any writing to a hard drive. I know very little about this, so it may not even be possible. I think it would be nice to have the dll data in the executable (like has been shown before in this thread), put those data into RAM (using a memoryblock, I guess), and use the dll from there. Is this possible?

I can't even imaging a way of calling the functions of the dll residing in memory, so I also thought that one could create a RAM-disk, create the dll there, and then use it normally employing the correct path.

So, would it be possible to use a dll stored in the RAM? If not, could it be possible to do it from a RAM-disk?

Pixe

_________________
Using RS2011r4.3 on Windows7.


Top
 Profile  
Reply with quote  
 Post subject: Re: MBS Plugins - Compression plugin
PostPosted: Fri Aug 26, 2011 5:30 pm 
Offline

Joined: Fri Jan 06, 2006 3:21 pm
Posts: 12388
Location: Portland, OR USA
That's the way RB *used to* handle dll's - load them directly into memory - back when they created a single .exe with no .dll lib folder. They abandoned the practice and went to on-disk dll files. So it can be done, but isn't recommended.


Top
 Profile  
Reply with quote  
 Post subject: Re: MBS Plugins - Compression plugin
PostPosted: Fri Aug 26, 2011 6:57 pm 
Offline
User avatar

Joined: Sun Aug 12, 2007 10:10 am
Posts: 1086
Location: Boiling Springs, SC
Could the dlls be embeded into the executable and written to a VirtualVolume and called from there? Afterall, vv's are flushed once the application ends.. So it would be just like loading them into memory without the wasted ram and proprietary dlls couldn't be "reused" by others ? Havnt tried it but definitely interested now.. I will attemp it and let you know my findings in a few minutes :-)

I love UPX!!!.. If you can find a previous version (not sure will have to look) it actually packs the executable even smaller..(so I dunno what they Did to the code to add the small bloat) Definitely a big help with RB apps and a must have :-) I wrote a utility that recursively scans my compiled projects and uses shell execute to compress the dll/exes :-) I wonder why RB hasn't thought of prepacking the dlls before distributing them? Maybe they don't know about UPX? .. Anyway some vv dll testing is underway..

_________________
Matthew A. Combatti
Real Studio 2012 r1.2

Visit Xojo Developer's Spot!
Systems I Use:
Windows XP/Windows Vista/Windows Server 2008 r2/Windows 7/Windows 8
Mac OSX 10.5/Mac OSX 10.6/Mac OSX Server/Ubuntu/Debian/Suse/Red Hat/
Windows Server 2011/CentOS 5.4 /ReactOS/SimOS

~All Xojo Compatible~


Top
 Profile  
Reply with quote  
 Post subject: Re: MBS Plugins - Compression plugin
PostPosted: Fri Aug 26, 2011 11:38 pm 
Offline
User avatar

Joined: Sun Aug 12, 2007 10:10 am
Posts: 1086
Location: Boiling Springs, SC
simulanics wrote:
Could the dlls be embeded into the executable and written to a VirtualVolume and called from there? Afterall, vv's are flushed once the application ends.. So it would be just like loading them into memory without the wasted ram and proprietary dlls couldn't be "reused" by others ? Havnt tried it but definitely interested now.. I will attemp it and let you know my findings in a few minutes :-)

I love UPX!!!.. If you can find a previous version (not sure will have to look) it actually packs the executable even smaller..(so I dunno what they Did to the code to add the small bloat) Definitely a big help with RB apps and a must have :-) I wrote a utility that recursively scans my compiled projects and uses shell execute to compress the dll/exes :-) I wonder why RB hasn't thought of prepacking the dlls before distributing them? Maybe they don't know about UPX? .. Anyway some vv dll testing is underway..



So after some monkeying around...I have discovered that you can infact embed a dll directly into your app (which I though was possible just didn't know how until now..so kudos to you for making us think!) but from your application..create a virtual volume and 'extract' the dll it... usually the dll MUST be in your system directory or same path as your application...UNLESS you specify the exact location to the dll

SEE this forum post for details...I had to search quite a bit but found it :-)
viewtopic.php?f=6&t=40290&p=224177&hilit=declare+lib#p224177

So yes, after compiling a little dll and packing it and testing it...it works..and it's much easier than loading it into memory and accessing it there...

_________________
Matthew A. Combatti
Real Studio 2012 r1.2

Visit Xojo Developer's Spot!
Systems I Use:
Windows XP/Windows Vista/Windows Server 2008 r2/Windows 7/Windows 8
Mac OSX 10.5/Mac OSX 10.6/Mac OSX Server/Ubuntu/Debian/Suse/Red Hat/
Windows Server 2011/CentOS 5.4 /ReactOS/SimOS

~All Xojo Compatible~


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 39 posts ]  Go to page Previous  1, 2, 3  Next

All times are UTC - 5 hours


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group