Author Topic: Couple Api ideas for embedding  (Read 90089 times)

dzzie

  • Guest
Couple Api ideas for embedding
« on: May 29, 2014, 06:30:01 PM »
Hi all,

Just found the project and I am really impressed with it. Before I start I will mention my goal. I would like to create a basic script control similar to MS Script control but with debugging / single step/ variable inspection support. So I am coming from the task of an embedded user.

ScriptBasic looks like it is a good fit for this and I have been experimenting with it for about a week now. So far I have added the following (simple) API

Code: [Select]
//text config to binary
scriba_CompileConfig

//add a preprocessor dll explicitly by dll path, no config needed
scriba_LoadInternalPreprocessorByPath

//register a compiled in preprocessor with no need for external dll
scriba_LoadInternalPreprocessorByFunction

I have also had success compiling extensions directly into the interpreter and accessing them via the declare mechanism (even when working from an exe test project) This seemed easier to me than writing a new command so far.

I see that the interpreter has support to call script functions and getting/setting global variables. I was looking for an easy way to have a host app register a new function to be available to scripts maybe something like

Code: [Select]
scriba_registerNative(pProgram, "myFunc", &myFunc_handler)

Thats when I took a side street and figured out to use a compiled in extension for this task.

Code: [Select]
declare sub trial alias "trial" lib "test.exe"

Bit of a hack but it works so I can move on to experiment with next task of wiring access to the now compiled in debugger preprocessor to a VB6 GUI. Once this is fleshed out, I can add in a scintilla based syntax highlight control I have and add proper debug controls around that.

Ultimate goal for me is to make an embeddable script control ocx with a script editor IDE + debug support and some kind of support for VB6 COM objects.

For COM support, I was thinking of something like:

Code: [Select]
scriba_registerNameSpaceResolver(pProgram, "myObj", &my_resolver)

so if the script engine sees myObj::myfunc(1,2)

it will call my_resolver() with the args "myfunc", 1, 2 or something like that, but after looking through the syntax and lexer code I think it would be tricky to slip this in. An easier solution is probably to use the extension trick to

Code: [Select]
declare sub callObj alias "callObj" lib "test.exe"
a = callObj("myObj.myFunc", 1 , 2)

Then the VB6 CallByName function will be put to good use, but you will be limited to accessing the primary functions/properties of the object I think. (ie myObj.itsObj.someProp probably wont work at least not  without some parsing and intermediate steps). Not the prettiest, but manageable.

John also mentioned Charles Pegge's (OxygenBasic author) DLLC COM extension which I will study as well.

Again really impressed with the project, I have wanted this type of script control for a loooong time. I can see all the pieces fitting together here.

My work in progress can be found here:  https://github.com/dzzie/ScriptBasic_Control

As always, any/all ideas or comments welcome :)

-Dave




Support

  • Administrator
  • *****
  • Posts: 19
    • View Profile
Re: Couple Api ideas for embedding
« Reply #1 on: May 30, 2014, 08:21:13 AM »
Welcome Dave!

Interesting stuff. Can you post your Visual Studio project file when you get everything working? It would be nice to have an alternative way to compile Script BASIC for Windows other than Peter's setup.pl method. I current have VS 2013 install on Win 7 64.



dzzie

  • Guest
Re: Couple Api ideas for embedding
« Reply #2 on: May 30, 2014, 12:26:35 PM »
the solution and project files are included in the git repository (also a .zip download link) https://github.com/dzzie/ScriptBasic_Control/archive/master.zip

My physical file layout is slightly different and does not include all files in the distro, but you can see the project settings used.
You could also remove the files I have set in there and re-add (just drag and drop a multiselection of files or a folder into the VS treeview and it will add all files in that dir to the project)

There are ways to add multiple projects to one solution file so you can "Build All" and it will compile each exe, dll , preprocessor, extension etc all at once.
Setting all that up might take a while though it can be allot of fussing. I think basically you make copies of a vcproj file you are using, set up each one individually to build the target you want, then from the top level solution file, you can "Add project". so each binary gets its own project file, all linked into
the top level solution file. Its kind of a pita.

Sometimes I just use a VS project just for code navigation and editing, then use the custom build scripts authors provide for the actual compilation. Actually there is probably a way to set a custom build step to use the perl script and not the standard VS compilation process. So you probably can just add all the files to one
solution file for editing/viewing/navigation, then hit build and it will call the perl script. Not sure if it will display compilation errors as nicely as normally though.
 

Support

  • Administrator
  • *****
  • Posts: 19
    • View Profile
Re: Couple Api ideas for embedding
« Reply #3 on: May 30, 2014, 12:37:48 PM »
For a release build, Peter's method would be preferred as everything (.h, syntax tables, error codes, documentation, ...) is built from scratch from the C files.

Armando (AIR) created a custom SB master make file for gcc under Windows that works pretty well.


dzzie

  • Guest
COM progress
« Reply #4 on: June 03, 2014, 07:45:15 AM »
thought I would share another bit of progress. I think a generic COM extension is about 90% done now and working in tests.
(currently i have it compiling into the main exe for debugging, no dll available yet but should be able to be built as one)

https://github.com/dzzie/ScriptBasic_Control/blob/master/engine/extensions/COM.cpp

Code: [Select]
declare sub CreateObject alias "CreateObject" lib "test.exe"
declare sub CallByName alias "CallByName" lib "test.exe"

const VbGet = 2
const VbLet = 4
const VbMethod = 1
const VbSet = 8

obj = CreateObject("SAPI.SpVoice")

if obj = 0 then
print "CreateObject failed!\n"
else
CallByName(obj, "rate", VbLet, 2)
CallByName(obj, "volume", VbLet, 60)
CallByName(obj, "speak", VbMethod, "This is my test")
end if

print "\n\nPress ENTER..."
line input a

'in vbscript this would be:
'----------------------------------------
'Set voice = CreateObject("SAPI.SpVoice")
'voice.Rate = 8
'voice.Volume = 60
'voice.Speak "Hello, world!"

Support

  • Administrator
  • *****
  • Posts: 19
    • View Profile
Re: COM progress
« Reply #5 on: June 03, 2014, 08:03:58 AM »
Dave,

Great progress! I'm looking forward to start using some of this code.

Do you think this would work with Mono as well? (COM & Mono)

Quote
Note that there is very little that ties COM Interop to Windows. COM is a binary standard, so anyone who follows that standard can use the COM Interop functionality in mono.

John
« Last Edit: June 03, 2014, 10:04:21 AM by support »

dzzie

  • Guest
Re: Couple Api ideas for embedding
« Reply #6 on: June 03, 2014, 04:07:09 PM »

Not sure I understand so I will kinda talk out loud.

Its true the COM standard can be implemented as a method of Interface and function pointer discovery on any system. Its just a communication protocol.

If the question is could this technique be used to call .NET code written in Mono, its possible I suppose. I know C# has a way of registering .NET classes for use with COM by using a COM visible attribute, creating a typelib, and registering itself in the registry. Going from COM to .NET and back feels like functionality that MS shoehorned in for compatibility reasons.

I have also seen an example where shellcode could initialize the .NET runtime and call code written in IL for execution. If i wanted to interact with .NET code I think I would look first at this technique because it probably does not require special registration of the .NET classes on the target system.

(this isnt the example i was thinking of but all i could find in a quick search)
http://blog.didierstevens.com/2010/04/13/net-shellcode/

I havent looked at these examples in depth but I would say being able to host the CLR in your code and let script basic call out to its methods should be doable from the shellcode example.

It comes down to having clients to talk to and how to you find them. On Windows COM is heavily registry dependent to locate the class containers (dlls)



Support

  • Administrator
  • *****
  • Posts: 19
    • View Profile
Re: COM progress
« Reply #7 on: June 03, 2014, 04:23:47 PM »
Dave,

Is this a 32 bit Windows only solution? IUP has a OCX container (ATL) that the webview control runs in. Maybe it could be used with your extension module.

Thanks for all the hard work you have put into this.

John

dzzie

  • Guest
Re: Couple Api ideas for embedding
« Reply #8 on: June 03, 2014, 04:56:39 PM »

I have been a high level COM user mostly, and have dealt with the low level details just a handful of times.

I believe you can write 64bit COM servers. COM also supports remote servers such as a machine across the network
and out of process servers like a seperate exe on same machine. Right now I have only specified it to create in process servers which means that only 32bit servers could be used if it was compiled in 32bit mode.

I have not tested if it compiles properly for x64. I will be using it for interacting with VB6 COM classes (dlls) which are exclusively
32 bit. My dev system is only 32 bit as well. I have been avoiding the WOW64 environment because of things like file and registry redirection. Since VB6 is limited to 32 bit only it just adds headache and no value.

As far as interacting with OCX controls. In order host a container you need to implemented a bunch of COM interfaces to give
it a place to display itself. I have no experience with these on a low level in C. Just calling methods on a COM object probably arent enough to display a GUI unless it has methods like CreateForm etc. OCX can also house regular COM classes as well so this is possible for self hosting I have a hexeditor control like this.

The experiments I am currently do with this are my cutting edge of what I can pull off lol. Which is what makes it fun :)
Cross platform dev, 32/64 compatiable code and low level COM are all new to me so no guarantees on what changes may be needed to get it to work outside of the environment I am building this for.

Support

  • Administrator
  • *****
  • Posts: 19
    • View Profile
Re: Couple Api ideas for embedding
« Reply #9 on: June 03, 2014, 05:08:25 PM »
The IUP ATL container takes care of all the display and initialization issues. It doesn't provide method and and event support. There is Lua COM support for the container but that is about it.
« Last Edit: June 03, 2014, 05:21:26 PM by support »

dzzie

  • Guest
Re: Couple Api ideas for embedding
« Reply #10 on: June 03, 2014, 05:21:32 PM »
If it has something like RegisterCallBack for you to set event handlers from passed in function pointers then it would be hopeful.

Support

  • Administrator
  • *****
  • Posts: 19
    • View Profile
Re: Couple Api ideas for embedding
« Reply #11 on: June 03, 2014, 05:25:20 PM »
Take a look and the IUP (PUI) SB extension module. All events go through a common message handler function. IupLoopStepWait() In DLLC the SB script functions are called directly. With my C extension module for IUP it returns back to SB on an event and i do an ICALL with the address of the script function. I have a set of C callback functions that IUP calls and sets global variables I reset after access in another call if needed.


« Last Edit: June 03, 2014, 05:31:43 PM by support »

Verhas

  • Guest
Re: Couple Api ideas for embedding
« Reply #12 on: June 05, 2014, 04:19:34 AM »
Wow this is impressive what you do with this interpreter. I am amazed.

As for the API suggestions my 2cents:

namespace lookup hooking is an interesting approach. The issue is that ScriptBasic was not designed to support namespaces. This is only a naming trick. The namespace:: prefix becomes part of the name and that is it.

Code: [Select]
scriba_registerNative(pProgram, "myFunc", &myFunc_handler)

is also an interesting idea, but it can actually be solved defining an external module. Note that the external module can be "part" of the embedding application. It does not need to be a DLL (or .so in UNIX). It can be statically linked to the application and be available to the interpreter so it can call the method. Note that the function pointers are defined in a table for the modules (as far as I remember, I have not touched it for 8 years now). In that table there is a

Code: [Select]
"myFunc" , &myFunc_handler

part that defines the function entry point and the associated name. It is true that you can not have different associations for different pProgram objects. It is static.


dzzie

  • Guest
Re: Couple Api ideas for embedding
« Reply #13 on: June 05, 2014, 06:55:35 PM »
Hi Peter

thanks for the ideas and the great code base !

 The internal extension is working well. Currently if more than one internal extension is compiled in besSUB_START and friends would conflict is only problem I see. I think this will be easy to work around by prepending module name to it.

Code: [Select]
#define besSUB_START  besFUNCTION(bootmodu)
#define MODULE_INITIALIZER "bootmodu"
/* modumana.c call the module initializer function */
FunctionPointer = modu_GetModuleFunctionByName(*ThisModule,MODULE_INITIALIZER);

so if functionPointer == NULL then maybe secondary check something like

Code: [Select]
sprintf(buf, "%s_bootmodu", (*ThisModule)->pszModuleName)
FunctionPointer = modu_GetModuleFunctionByName(*ThisModule,buf);

seems like should work. actually in further thought..maybe it wont I was thinking of preprocessor name, pszModuleName is probably taken from declare i havent step breakpoint yet to check for sure.  (also forgot to mention above I also had to disable strcat for dll extension to module name for it to work from exe.) Some variation of this concept will work anyway though.

I also found a good article on x64 COM.

http://mariusbancila.ro/blog/2010/11/04/32-bit-and-64-bit-com-servers/

Have not had a chance to look at the IUP yet except glance. Looking into IConnectionPoint maybe interesting experiment to see about how to connect call back events from COM back to SB event handlers something like

Code: [Select]
'sb code
ConnectEvents(objPtr, "objName")

sub objName_Clicked

then in C I imagine it works something like take objPtr, enumerate events object supports
then check script to see if objName_[event name] exists..if it does then handle the event and
call the sub manually when its hit. Interesting to think of the logistics of it anyway. Sinking events
may be required for IUP OCX as well from sounds of it.

More time to play on the weekend.

One thing I am not sure on is if I want to give script clients live object handles for them to passback
and me use blindly. Its kind of a bad idea. try catch and is null checks can handle some ok, but a random number
passed in could easily lead to memory corruption even if try catch kept execution going for now. Safer would be to
only pass back quasi handles I suppose and then validate them from an internal hash map to real value.

embedded clients who do things like AddObject to make object available to script would have to add to list as well.
Only down side is if a COM object returned another COM object, the script would not be able to use it now since
the manager would not be aware of the pointer being valid. (or script could be given access to register its own then
stability is all in their hands)

anyway i think the COM extension will be left for a while now. majority is done, will need to build out the other components
to circle back and start really using it to see what more it needs. One surprising thing, if a vb com object expects a byte type (VT_I1), it will accept a VT_I4 and .lvalue as long as value < 255. I suppose not so surprising since I think the value is a union, but still seems like it is being kind with an auto conversion accepting different type in background. I will have to test some ATL based clients to see if they are as kind.

but so far so good :)



« Last Edit: June 05, 2014, 07:23:40 PM by dzzie »

Support

  • Administrator
  • *****
  • Posts: 19
    • View Profile
Re: Couple Api ideas for embedding
« Reply #14 on: June 05, 2014, 07:43:31 PM »
Great progress Dave!

Let me know if you need some help with testing.

Quote
Currently if more than one internal extension is compiled in besSUB_START and friends would conflict is only problem I see.

Each internal extension should be compiled into its own object library. I assume SB skips the whole DLL handshacking routine knowing that it's static linked. (no load/unload stuff going on or version checking needed) Peter would have to validate my understanding if it's true or not.

« Last Edit: June 05, 2014, 11:46:51 PM by support »